Order Matters: Agent-by-agent Policy Optimization

Abstract

While multi-agent trust region algorithms have achieved great success empirically in solving coordination tasks, most of them, however, suffer from a non-stationarity problem since agents update their policies simultaneously. In contrast, a sequential scheme that updates policies agent-by-agent provides another perspective and shows strong performance. However, sample inefficiency and lack of monotonic improvement guarantees for each agent are still the two significant challenges for the sequential scheme. In this paper, we propose the \textbf\{A\}gent-by-\textbf\{a\}gent \textbf\{P\}olicy \textbf\{O\}ptimization (A2PO) algorithm to improve the sample efficiency and retain the guarantees of monotonic improvement for each agent during training. We justify the tightness of the monotonic improvement bound compared with other trust region algorithms. From the perspective of sequentially updating agents, we further consider the effect of agent updating order and extend the theory of non-sta

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2023order

Related papers

• B2MAPO: A Batch-by-batch Multi-agent Policy Optimization To Balance Performance And Efficiency (2024)0.00
• Trust Region Policy Optimisation In Multi-agent Reinforcement Learning (2021)0.00
• Multi-agent Trust Region Policy Optimization (2020)12.61
• Trust Region Bounds For Decentralized PPO Under Non-stationarity (2022)0.00
• Local Optimization Achieves Global Optimality In Multi-agent Reinforcement Learning (2023)0.00
• An Analytical Update Rule For General Policy Optimization (2021)0.00
• Simple Policy Optimization (2024)0.00
• Policy Optimization For Markov Games: Unified Framework And Faster Convergence (2022)0.00