Alberdice: Addressing Out-of-distribution Joint Actions In Offline Multi-agent RL Via Alternating Stationary Distribution Correction Estimation

Abstract

One of the main challenges in offline Reinforcement Learning (RL) is the distribution shift that arises from the learned policy deviating from the data collection policy. This is often addressed by avoiding out-of-distribution (OOD) actions during policy improvement as their presence can lead to substantial performance degradation. This challenge is amplified in the offline Multi-Agent RL (MARL) setting since the joint action space grows exponentially with the number of agents. To avoid this curse of dimensionality, existing MARL methods adopt either value decomposition methods or fully decentralized training of individual agents. However, even when combined with standard conservatism principles, these methods can still result in the selection of OOD joint actions in offline MARL. To this end, we introduce AlberDICE, an offline MARL algorithm that alternatively performs centralized training of individual agents based on stationary distribution optimization. AlberDICE circumvents the ex

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymatsunaga2023alberdice

Related papers

• Comadice: Offline Cooperative Multi-agent Reinforcement Learning With Stationary Distribution Shift Regularization (2024)0.00
• Optidice: Offline Policy Optimization Via Stationary Distribution Correction Estimation (2021)0.00
• Coptidice: Offline Constrained Reinforcement Learning Via Stationary Distribution Correction Estimation (2022)0.00
• Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
• ODICE: Revealing The Mystery Of Distribution Correction Estimation Via Orthogonal-gradient Update (2024)0.00
• Offline Multi-agent Reinforcement Learning Via In-sample Sequential Policy Optimization (2024)0.00
• Counterfactual Conservative Q Learning For Offline Multi-agent Reinforcement Learning (2023)0.00
• Offline Reinforcement Learning With OOD State Correction And OOD Action Suppression (2024)0.00