Heterogeneous-agent Reinforcement Learning

Abstract

The necessity for cooperation among intelligent machines has popularised cooperative multi-agent reinforcement learning (MARL) in AI research. However, many research endeavours heavily rely on parameter sharing among agents, which confines them to only homogeneous-agent setting and leads to training instability and lack of convergence guarantees. To achieve effective cooperation in the general heterogeneous-agent setting, we propose Heterogeneous-Agent Reinforcement Learning (HARL) algorithms that resolve the aforementioned issues. Central to our findings are the multi-agent advantage decomposition lemma and the sequential update scheme. Based on these, we develop the provably correct Heterogeneous-Agent Trust Region Learning (HATRL), and derive HATRPO and HAPPO by tractable approximations. Furthermore, we discover a novel framework named Heterogeneous-Agent Mirror Learning (HAML), which strengthens theoretical guarantees for HATRPO and HAPPO and provides a general template for coopera

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhong2023heterogeneous

Related papers

• Heterogeneous-agent Mirror Learning: A Continuum Of Solutions To Cooperative MARL (2022)0.00
• Maximum Entropy Heterogeneous-agent Reinforcement Learning (2023)0.00
• Trust Region Policy Optimisation In Multi-agent Reinforcement Learning (2021)0.00
• Heterogeneous Multi-agent Reinforcement Learning Via Mirror Descent Policy Optimization (2023)0.00
• Heterogeneous Multi-agent Reinforcement Learning For Zero-shot Scalable Collaboration (2024)6.34
• Heterogeneous Multi-robot Reinforcement Learning (2023)6.77
• Learning Heterogeneous Agent Cooperation Via Multiagent League Training (2022)7.16
• GHQ: Grouped Hybrid Q Learning For Heterogeneous Cooperative Multi-agent Reinforcement Learning (2023)6.34