Mixed Q-functionals: Advancing Value-based Methods In Cooperative MARL With Continuous Action Domains

Abstract

Tackling multi-agent learning problems efficiently is a challenging task in continuous action domains. While value-based algorithms excel in sample efficiency when applied to discrete action domains, they are usually inefficient when dealing with continuous actions. Policy-based algorithms, on the other hand, attempt to address this challenge by leveraging critic networks for guiding the learning process and stabilizing the gradient estimation. The limitations in the estimation of true return and falling into local optima in these methods result in inefficient and often sub-optimal policies. In this paper, we diverge from the trend of further enhancing critic networks, and focus on improving the effectiveness of value-based methods in multi-agent continuous domains by concurrently evaluating numerous actions. We propose a novel multi-agent value-based algorithm, Mixed Q-Functionals (MQF), inspired from the idea of Q-Functionals, that enables agents to transform their states into basis

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfindik2024mixed

Related papers

• Residual Q-networks For Value Function Factorizing In Multi-agent Reinforcement Learning (2022)10.21
• Uneven: Universal Value Exploration For Multi-agent Reinforcement Learning (2020)0.00
• QMIX: Monotonic Value Function Factorisation For Deep Multi-agent Reinforcement Learning (2018)0.00
• NQMIX: Non-monotonic Value Function Factorization For Deep Multi-agent Reinforcement Learning (2021)0.00
• Weighted QMIX: Expanding Monotonic Value Function Factorisation For Deep Multi-agent Reinforcement Learning (2020)0.00
• Monotonic Value Function Factorisation For Deep Multi-agent Reinforcement Learning (2020)0.00
• Policy Distillation And Value Matching In Multiagent Reinforcement Learning (2019)10.48
• Qatten: A General Framework For Cooperative Multiagent Reinforcement Learning (2020)0.00