Factorized Q-learning For Large-scale Multi-agent Systems

Abstract

Deep Q-learning has achieved significant success in single-agent decision making tasks. However, it is challenging to extend Q-learning to large-scale multi-agent scenarios, due to the explosion of action space resulting from the complex dynamics between the environment and the agents. In this paper, we propose to make the computation of multi-agent Q-learning tractable by treating the Q-function (w.r.t. state and joint-action) as a high-order high-dimensional tensor and then approximate it with factorized pairwise interactions. Furthermore, we utilize a composite deep neural network architecture for computing the factorized Q-function, share the model parameters among all the agents within the same group, and estimate the agents' optimal joint actions through a coordinate descent type algorithm. All these simplifications greatly reduce the model complexity and accelerate the learning process. Extensive experiments on two different multi-agent problems demonstrate the performance gain

Stats

• citations34
• S2 citations—
• github stars0
• HF likes0
• heat score11.58
• arxiv keyzhou2018factorized

Related papers

• Towards Understanding Cooperative Multi-agent Q-learning With Value Factorization (2020)0.00
• Multi-agent Determinantal Q-learning (2020)0.00
• Residual Q-networks For Value Function Factorizing In Multi-agent Reinforcement Learning (2022)10.21
• Analysing Factorizations Of Action-value Networks For Cooperative Multi-agent Reinforcement Learning (2019)2.26
• Concaveq: Non-monotonic Value Function Factorization Via Concave Representations In Deep Multi-agent Reinforcement Learning (2023)5.84
• High-order Interactions Modeling For Interpretable Multi-agent Q-learning (2025)0.00
• A Finite Time Analysis Of Distributed Q-learning (2024)0.00
• Qfree: A Universal Value Function Factorization For Multi-agent Reinforcement Learning (2023)0.00