Cooperative Multi-agent Reinforcement Learning With Partial Observations

Abstract

In this paper, we propose a distributed zeroth-order policy optimization method for Multi-Agent Reinforcement Learning (MARL). Existing MARL algorithms often assume that every agent can observe the states and actions of all the other agents in the network. This can be impractical in large-scale problems, where sharing the state and action information with multi-hop neighbors may incur significant communication overhead. The advantage of the proposed zeroth-order policy optimization method is that it allows the agents to compute the local policy gradients needed to update their local policy functions using local estimates of the global accumulated rewards that depend on partial state and action information only and can be obtained using consensus. Specifically, to calculate the local policy gradients, we develop a new distributed zeroth-order policy gradient estimator that relies on one-point residual-feedback which, compared to existing zeroth-order estimators that also rely on one-poi

Stats

• citations23
• S2 citations—
• github stars0
• HF likes0
• heat score10.35
• arxiv keyzhang2020cooperative

Related papers

• Faster Last-iterate Convergence Of Policy Optimization In Zero-sum Markov Games (2022)0.00
• Scalable And Sample Efficient Distributed Policy Gradient Algorithms In Multi-agent Networked Systems (2022)0.00
• Multi-agent Trust Region Policy Optimization (2020)12.61
• Fully Decentralized Multi-agent Reinforcement Learning With Networked Agents (2018)0.00
• Distributed Policy Gradient With Variance Reduction In Multi-agent Reinforcement Learning (2021)0.00
• Offline Multi-agent Reinforcement Learning Via In-sample Sequential Policy Optimization (2024)0.00
• Multi-agent Reinforcement Learning In Stochastic Networked Systems (2020)0.00
• Mean-field Multi-agent Reinforcement Learning: A Decentralized Network Approach (2021)0.00