Multi-agent Reinforcement Learning Via Double Averaging Primal-dual Optimization

Abstract

Despite the success of single-agent reinforcement learning, multi-agent reinforcement learning (MARL) remains challenging due to complex interactions between agents. Motivated by decentralized applications such as sensor networks, swarm robotics, and power grids, we study policy evaluation in MARL, where agents with jointly observed state-action pairs and private local rewards collaborate to learn the value of a given policy. In this paper, we propose a double averaging scheme, where each agent iteratively performs averaging over both space and time to incorporate neighboring gradient information and local reward information, respectively. We prove that the proposed algorithm converges to the optimal solution at a global geometric rate. In particular, such an algorithm is built upon a primal-dual reformulation of the mean squared projected Bellman error minimization problem, which gives rise to a decentralized convex-concave saddle-point problem. To the best of our knowledge, the propo

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywai2018multi

Related papers

• Cooperative Multi-agent Reinforcement Learning With Partial Observations (2020)10.35
• Faster Last-iterate Convergence Of Policy Optimization In Zero-sum Markov Games (2022)0.00
• Multi-agent Reinforcement Learning In Stochastic Networked Systems (2020)0.00
• MARL With General Utilities Via Decentralized Shadow Reward Actor-critic (2021)0.00
• Policy Distillation And Value Matching In Multiagent Reinforcement Learning (2019)10.48
• Scalable Multi-agent Reinforcement Learning For Networked Systems With Average Reward (2020)0.00
• Revisiting Some Common Practices In Cooperative Multi-agent Reinforcement Learning (2022)0.00
• Decentralized Multi-agent Reinforcement Learning With Networked Agents: Recent Advances (2019)0.00