A Variational Approach To Mutual Information-based Coordination For Multi-agent Reinforcement Learning

Abstract

In this paper, we propose a new mutual information framework for multi-agent reinforcement learning to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the simultaneous mutual information between multi-agent actions. By introducing a latent variable to induce nonzero mutual information between multi-agent actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function. The derived tractable objective can be interpreted as maximum entropy reinforcement learning combined with uncertainty reduction of other agents actions. Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic, which follows centralized learning with decentralized execution. We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms other MARL algo

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykim2023a

Related papers

• Learning To Coordinate In Multi-agent Systems: A Coordinated Actor-critic Algorithm And Finite-time Guarantees (2021)0.00
• Robust Multi-agent Reinforcement Learning By Mutual Information Regularization (2023)0.00
• Iterated Reasoning With Mutual Information In Cooperative And Byzantine Decentralized Teaming (2022)0.00
• NVIF: Neighboring Variational Information Flow For Large-scale Cooperative Multi-agent Scenarios (2022)0.00
• Bi-level Actor-critic For Multi-agent Coordination (2019)0.00
• PMIC: Improving Multi-agent Reinforcement Learning With Progressive Mutual Information Collaboration (2022)0.00
• Efficient Communication In Multi-agent Reinforcement Learning Via Variance Based Control (2019)0.00
• Maximum Entropy Heterogeneous-agent Reinforcement Learning (2023)0.00