Major-minor Mean Field Multi-agent Reinforcement Learning

Abstract

Multi-agent reinforcement learning (MARL) remains difficult to scale to many agents. Recent MARL using Mean Field Control (MFC) provides a tractable and rigorous approach to otherwise difficult cooperative MARL. However, the strict MFC assumption of many independent, weakly-interacting agents is too inflexible in practice. We generalize MFC to instead simultaneously model many similar and few complex agents -- as Major-Minor Mean Field Control (M3FC). Theoretically, we give approximation results for finite agent control, and verify the sufficiency of stationary policies for optimality together with a dynamic programming principle. Algorithmically, we propose Major-Minor Mean Field MARL (M3FMARL) for finite agent systems instead of the limiting system. The algorithm is shown to approximate the policy gradient of the underlying M3FC MDP. Finally, we demonstrate its capabilities experimentally in various scenarios. We observe a strong performance in comparison to state-of-the-art policy g

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycui2023major

Related papers

• Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) With Non-uniform Interaction? (2022)0.00
• Mean-field Approximation Of Cooperative Constrained Multi-agent Reinforcement Learning (CMARL) (2022)0.00
• Mean-field Control Based Approximation Of Multi-agent Reinforcement Learning In Presence Of A Non-decomposable Shared Global State (2023)0.00
• On The Approximation Of Cooperative Heterogeneous Multi-agent Reinforcement Learning (MARL) Using Mean Field Control (MFC) (2021)0.00
• Efficient Model-based Multi-agent Mean-field Reinforcement Learning (2021)0.00
• Efficient And Scalable Deep Reinforcement Learning For Mean Field Control Games (2024)0.00
• Multi Type Mean Field Reinforcement Learning (2020)0.00
• Model-free Mean-field Reinforcement Learning: Mean-field MDP And Mean-field Q-learning (2019)0.00