On Centralized Critics In Multi-agent Reinforcement Learning

Abstract

Centralized Training for Decentralized Execution where agents are trained offline in a centralized fashion and execute online in a decentralized manner, has become a popular approach in Multi-Agent Reinforcement Learning (MARL). In particular, it has become popular to develop actor-critic methods that train decentralized actors with a centralized critic where the centralized critic is allowed access global information of the entire system, including the true system state. Such centralized critics are possible given offline information and are not used for online execution. While these methods perform well in a number of domains and have become a de facto standard in MARL, using a centralized critic in this context has yet to be sufficiently analyzed theoretically or empirically. In this paper, we therefore formally analyze centralized and decentralized critic approaches, and analyze the effect of using state-based critics in partially observable environments. We derive theories contrar

Stats

• citations15
• S2 citations—
• github stars0
• HF likes0
• heat score9.03
• arxiv keylyu2024on

Related papers

• Contrasting Centralized And Decentralized Critics In Multi-agent Reinforcement Learning (2021)0.00
• A Deeper Understanding Of State-based Critics In Multi-agent Reinforcement Learning (2022)6.34
• Actor-attention-critic For Multi-agent Reinforcement Learning (2018)0.00
• Fully Decentralized Multi-agent Reinforcement Learning With Networked Agents (2018)0.00
• F2A2: Flexible Fully-decentralized Approximate Actor-critic For Cooperative Multi-agent Reinforcement Learning (2020)0.00
• Communication-efficient Actor-critic Methods For Homogeneous Markov Games (2022)0.00
• Is Centralized Training With Decentralized Execution Framework Centralized Enough For MARL? (2023)0.00
• Reducing Overestimation Bias In Multi-agent Domains Using Double Centralized Critics (2019)0.00