Reinforcement Learning In Linear Quadratic Deep Structured Teams: Global Convergence Of Policy Gradient Methods

Abstract

In this paper, we study the global convergence of model-based and model-free policy gradient descent and natural policy gradient descent algorithms for linear quadratic deep structured teams. In such systems, agents are partitioned into a few sub-populations wherein the agents in each sub-population are coupled in the dynamics and cost function through a set of linear regressions of the states and actions of all agents. Every agent observes its local state and the linear regressions of states, called deep states. For a sufficiently small risk factor and/or sufficiently large population, we prove that model-based policy gradient methods globally converge to the optimal solution. Given an arbitrary number of agents, we develop model-free policy gradient and natural policy gradient algorithms for the special case of risk-neutral cost function. The proposed algorithms are scalable with respect to the number of agents due to the fact that the dimension of their policy space is independent o

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyfathi2020reinforcement

Related papers

• Reinforcement Learning In Nonzero-sum Linear Quadratic Deep Structured Games: Global Convergence Of Policy Optimization (2020)6.77
• Linear-quadratic Mean-field Reinforcement Learning: Convergence Of Policy Gradient Methods (2019)0.00
• Global Convergence Using Policy Gradient Methods For Model-free Markovian Jump Linear Quadratic Control (2021)0.00
• Global Convergence Of Policy Gradient For Linear-quadratic Mean-field Control/game In Continuous Time (2020)0.00
• On The Linear Convergence Of Natural Policy Gradient Algorithm (2021)0.00
• Global Convergence Of Policy Gradient Methods In Reinforcement Learning, Games And Control (2023)0.00
• Dimension-free Rates For Natural Policy Gradient In Multi-agent Reinforcement Learning (2021)0.00
• Policy-gradient Algorithms Have No Guarantees Of Convergence In Linear Quadratic Games (2019)5.24