Locality Matters: A Scalable Value Decomposition Approach For Cooperative Multi-agent Reinforcement Learning

Abstract

Cooperative multi-agent reinforcement learning (MARL) faces significant scalability issues due to state and action spaces that are exponentially large in the number of agents. As environments grow in size, effective credit assignment becomes increasingly harder and often results in infeasible learning times. Still, in many real-world settings, there exist simplified underlying dynamics that can be leveraged for more scalable solutions. In this work, we exploit such locality structures effectively whilst maintaining global cooperation. We propose a novel, value-based multi-agent algorithm called LOMAQ, which incorporates local rewards in the Centralized Training Decentralized Execution paradigm. Additionally, we provide a direct reward decomposition method for finding these local rewards when only a global signal is provided. We test our method empirically, showing it scales well compared to other methods, significantly improving performance and convergence speed.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzohar2021locality

Related papers

• Local Advantage Networks For Cooperative Multi-agent Reinforcement Learning (2021)0.00
• Scalable Multi-agent Reinforcement Learning For Networked Systems With Average Reward (2020)0.00
• Adaptive Value Decomposition With Greedy Marginal Contribution Computation For Cooperative Multi-agent Reinforcement Learning (2023)3.58
• Q-value Path Decomposition For Deep Multiagent Reinforcement Learning (2020)0.00
• Multi-agent Reinforcement Learning In Stochastic Networked Systems (2020)0.00
• Revisiting Some Common Practices In Cooperative Multi-agent Reinforcement Learning (2022)0.00
• MARL-LNS: Cooperative Multi-agent Reinforcement Learning Via Large Neighborhoods Search (2024)0.00
• Understanding Value Decomposition Algorithms In Deep Cooperative Multi-agent Reinforcement Learning (2022)0.00