Offline Multi-agent Reinforcement Learning With Implicit Global-to-local Value Regularization

Abstract

Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline m ulti-agent RL algorithm with implicit global-to-local v alue regularization. OMIGA provides a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning wi

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keywang2023offline

Related papers

• Offline-to-online Multi-agent Reinforcement Learning With Offline Value Function Memory And Sequential Exploration (2024)2.26
• Offline Multi-agent Reinforcement Learning Via In-sample Sequential Policy Optimization (2024)0.00
• Comadice: Offline Cooperative Multi-agent Reinforcement Learning With Stationary Distribution Shift Regularization (2024)0.00
• FOCAL: Efficient Fully-offline Meta-reinforcement Learning Via Distance Metric Learning And Behavior Regularization (2020)0.00
• Offline RL With No OOD Actions: In-sample Learning Via Implicit Value Regularization (2023)0.00
• Locality Matters: A Scalable Value Decomposition Approach For Cooperative Multi-agent Reinforcement Learning (2021)0.00
• Offline Decentralized Multi-agent Reinforcement Learning (2021)7.50
• Believe What You See: Implicit Constraint Approach For Offline Multi-agent Reinforcement Learning (2021)0.00