Bi-level Actor-critic For Multi-agent Coordination

Abstract

Coordination is one of the essential problems in multi-agent systems. Typically multi-agent reinforcement learning (MARL) methods treat agents equally and the goal is to solve the Markov game to an arbitrary Nash equilibrium (NE) when multiple equilibra exist, thus lacking a solution for NE selection. In this paper, we treat agents *unequally* and consider Stackelberg equilibrium as a potentially better convergence point than Nash equilibrium in terms of Pareto superiority, especially in cooperative environments. Under Markov games, we formally define the bi-level reinforcement learning problem in finding Stackelberg equilibrium. We propose a novel bi-level actor-critic learning method that allows agents to have different knowledge base (thus intelligent), while their actions still can be executed simultaneously and distributedly. The convergence proof is given, while the resulting learning algorithm is tested against the state of the arts. We found that the proposed bi-level actor-cri

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2019bi

Related papers

• Multi-agent Actor-critic For Mixed Cooperative-competitive Environments (2017)0.00
• Learning To Coordinate In Multi-agent Systems: A Coordinated Actor-critic Algorithm And Finite-time Guarantees (2021)0.00
• Context-aware Bayesian Network Actor-critic Methods For Cooperative Multi-agent Reinforcement Learning (2023)0.00
• Actor-attention-critic For Multi-agent Reinforcement Learning (2018)0.00
• Actor-critic Algorithms For Constrained Multi-agent Reinforcement Learning (2019)0.00
• Equilibrium Selection For Multi-agent Reinforcement Learning: A Unified Framework (2024)0.00
• Communication-efficient Actor-critic Methods For Homogeneous Markov Games (2022)0.00
• Stackelberg Actor-critic: Game-theoretic Reinforcement Learning Algorithms (2021)0.00