Smix(\(\lambda\)): Enhancing Centralized Value Functions For Cooperative Multi-agent Reinforcement Learning

Abstract

Learning a stable and generalizable centralized value function (CVF) is a crucial but challenging task in multi-agent reinforcement learning (MARL), as it has to deal with the issue that the joint action space increases exponentially with the number of agents in such scenarios. This paper proposes an approach, named SMIX(\(\{\lambda\}\)), to address the issue using an efficient off-policy centralized training method within a flexible learner search space. As importance sampling for such off-policy training is both computationally costly and numerically unstable, we proposed to use the \(\{\lambda\}\)-return as a proxy to compute the TD error. With this new loss function objective, we adopt a modified QMIX network structure as the base to train our model. By further connecting it with the \(\{Q(\lambda)\}\) approach from an unified expectation correction viewpoint, we show that the proposed SMIX(\(\{\lambda\}\)) is equivalent to \(\{Q(\lambda)\}\) and hence shares its convergence proper

Stats

• citations13
• S2 citations—
• github stars0
• HF likes0
• heat score8.60
• arxiv keyyao2019smix

Related papers

• QMIX: Monotonic Value Function Factorisation For Deep Multi-agent Reinforcement Learning (2018)0.00
• QR-MIX: Distributional Value Function Factorisation For Cooperative Multi-agent Reinforcement Learning (2020)0.00
• Weighted QMIX: Expanding Monotonic Value Function Factorisation For Deep Multi-agent Reinforcement Learning (2020)0.00
• Monotonic Value Function Factorisation For Deep Multi-agent Reinforcement Learning (2020)0.00
• RMIX: Learning Risk-sensitive Policies For Cooperative Reinforcement Learning Agents (2021)0.00
• Uneven: Universal Value Exploration For Multi-agent Reinforcement Learning (2020)0.00
• NQMIX: Non-monotonic Value Function Factorization For Deep Multi-agent Reinforcement Learning (2021)0.00
• MMD-MIX: Value Function Factorisation With Maximum Mean Discrepancy For Cooperative Multi-agent Reinforcement Learning (2021)0.00