Global Convergence Of Localized Policy Iteration In Networked Multi-agent Reinforcement Learning

Abstract

We study a multi-agent reinforcement learning (MARL) problem where the agents interact over a given network. The goal of the agents is to cooperatively maximize the average of their entropy-regularized long-term rewards. To overcome the curse of dimensionality and to reduce communication, we propose a Localized Policy Iteration (LPI) algorithm that provably learns a near-globally-optimal policy using only local information. In particular, we show that, despite restricting each agent's attention to only its \(\kappa\)-hop neighborhood, the agents are able to learn a policy with an optimality gap that decays polynomially in \(\kappa\). In addition, we show the finite-sample convergence of LPI to the global optimal policy, which explicitly captures the trade-off between optimality and computational complexity in choosing \(\kappa\). Numerical simulations demonstrate the effectiveness of LPI.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyzhang2022global

Related papers

• Multi-agent Reinforcement Learning In Stochastic Networked Systems (2020)0.00
• Fully Decentralized Multi-agent Reinforcement Learning With Networked Agents (2018)0.00
• Scalable Multi-agent Reinforcement Learning For Networked Systems With Average Reward (2020)0.00
• Faster Last-iterate Convergence Of Policy Optimization In Zero-sum Markov Games (2022)0.00
• Local Advantage Networks For Cooperative Multi-agent Reinforcement Learning (2021)0.00
• Local Optimization Achieves Global Optimality In Multi-agent Reinforcement Learning (2023)0.00
• Learning To Share In Multi-agent Reinforcement Learning (2021)0.00
• Scalable And Sample Efficient Distributed Policy Gradient Algorithms In Multi-agent Networked Systems (2022)0.00