Divergence-regularized Multi-agent Actor-critic

Abstract

Entropy regularization is a popular method in reinforcement learning (RL). Although it has many advantages, it alters the RL objective of the original Markov Decision Process (MDP). Though divergence regularization has been proposed to settle this problem, it cannot be trivially applied to cooperative multi-agent reinforcement learning (MARL). In this paper, we investigate divergence regularization in cooperative MARL and propose a novel off-policy cooperative MARL framework, divergence-regularized multi-agent actor-critic (DMAC). Theoretically, we derive the update rule of DMAC which is naturally off-policy and guarantees monotonic policy improvement and convergence in both the original MDP and divergence-regularized MDP. We also give a bound of the discrepancy between the converged policy and optimal policy in the original MDP. DMAC is a flexible framework and can be combined with many existing MARL algorithms. Empirically, we evaluate DMAC in a didactic stochastic game and StarCraft

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysu2021divergence

Related papers

• Mirror Descent Actor Critic Via Bounded Advantage Learning (2025)0.00
• Mutual-information Regularization In Markov Decision Processes And Actor-critic Learning (2019)0.00
• MARL With General Utilities Via Decentralized Shadow Reward Actor-critic (2021)0.00
• Regularize! Don't Mix: Multi-agent Reinforcement Learning Without Explicit Centralized Structures (2021)0.00
• Diversity Actor-critic: Sample-aware Entropy Regularization For Sample-efficient Exploration (2020)0.00
• Maximum Entropy Heterogeneous-agent Reinforcement Learning (2023)0.00
• Robust Multi-agent Reinforcement Learning Via Adversarial Regularization: Theoretical Foundation And Stable Algorithms (2023)2.98
• Entropic Regularization Of Markov Decision Processes (2019)6.77