Contrasting Centralized And Decentralized Critics In Multi-agent Reinforcement Learning

Abstract

Centralized Training for Decentralized Execution, where agents are trained offline using centralized information but execute in a decentralized manner online, has gained popularity in the multi-agent reinforcement learning community. In particular, actor-critic methods with a centralized critic and decentralized actors are a common instance of this idea. However, the implications of using a centralized critic in this context are not fully discussed and understood even though it is the standard choice of many algorithms. We therefore formally analyze centralized and decentralized critic approaches, providing a deeper understanding of the implications of critic choice. Because our theory makes unrealistic assumptions, we also empirically compare the centralized and decentralized critic methods over a wide set of environments to validate our theories and to provide practical advice. We show that there exist misconceptions regarding centralized critics in the current literature and show th

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylyu2021contrasting

Related papers

• On Centralized Critics In Multi-agent Reinforcement Learning (2024)9.03
• A Deeper Understanding Of State-based Critics In Multi-agent Reinforcement Learning (2022)6.34
• Actor-attention-critic For Multi-agent Reinforcement Learning (2018)0.00
• Reducing Overestimation Bias In Multi-agent Domains Using Double Centralized Critics (2019)0.00
• F2A2: Flexible Fully-decentralized Approximate Actor-critic For Cooperative Multi-agent Reinforcement Learning (2020)0.00
• Multi-agent Natural Actor-critic Reinforcement Learning Algorithms (2021)3.58
• Communication-efficient Actor-critic Methods For Homogeneous Markov Games (2022)0.00
• Fully Decentralized Multi-agent Reinforcement Learning With Networked Agents (2018)0.00