Comadice: Offline Cooperative Multi-agent Reinforcement Learning With Stationary Distribution Shift Regularization

Abstract

Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we introduce a regularizer in the space of stationary distributions to better handle distributional shif

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybui2024comadice

Related papers

• Alberdice: Addressing Out-of-distribution Joint Actions In Offline Multi-agent RL Via Alternating Stationary Distribution Correction Estimation (2023)0.00
• Bridging Distributionally Robust Learning And Offline RL: An Approach To Mitigate Distribution Shift And Partial Data Coverage (2023)0.00
• Mitigating Distribution Shift In Model-based Offline RL Via Shifts-aware Reward Learning (2024)0.00
• Regularizing A Model-based Policy Stationary Distribution To Stabilize Offline Reinforcement Learning (2022)0.00
• State-constrained Offline Reinforcement Learning (2024)0.00
• Offline Multi-agent Reinforcement Learning With Implicit Global-to-local Value Regularization (2023)5.84
• Regularize! Don't Mix: Multi-agent Reinforcement Learning Without Explicit Centralized Structures (2021)0.00
• Optidice: Offline Policy Optimization Via Stationary Distribution Correction Estimation (2021)0.00