ACE: Cooperative Multi-agent Q-learning With Bidirectional Action-dependency

Abstract

Multi-agent reinforcement learning (MARL) suffers from the non-stationarity problem, which is the ever-changing targets at every iteration when multiple agents update their policies at the same time. Starting from first principle, in this paper, we manage to solve the non-stationarity problem by proposing bidirectional action-dependent Q-learning (ACE). Central to the development of ACE is the sequential decision-making process wherein only one agent is allowed to take action at one time. Within this process, each agent maximizes its value function given the actions taken by the preceding agents at the inference stage. In the learning phase, each agent minimizes the TD error that is dependent on how the subsequent agents have reacted to their chosen action. Given the design of bidirectional dependency, ACE effectively turns a multiagent MDP into a single-agent MDP. We implement the ACE framework by identifying the proper network representation to formulate the action dependency, so tha

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2022ace

Related papers

• Multi-agent Advisor Q-learning (2021)0.00
• Dealing With Non-stationarity In Decentralized Cooperative Multi-agent Deep Reinforcement Learning Via Multi-timescale Learning (2023)0.00
• Context-aware Bayesian Network Actor-critic Methods For Cooperative Multi-agent Reinforcement Learning (2023)0.00
• MA2QL: A Minimalist Approach To Fully Decentralized Multi-agent Reinforcement Learning (2022)0.00
• Q-value Path Decomposition For Deep Multiagent Reinforcement Learning (2020)0.00
• Inducing Cooperation Via Team Regret Minimization Based Multi-agent Deep Reinforcement Learning (2019)0.00
• Cooperative And Competitive Biases For Multi-agent Reinforcement Learning (2021)2.26
• Hierarchical Deep Multiagent Reinforcement Learning With Temporal Abstraction (2018)0.00