Offline Multi-agent Reinforcement Learning Via In-sample Sequential Policy Optimization

Abstract

Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2024offline

Related papers

• Learning From Good Trajectories In Offline Multi-agent Reinforcement Learning (2022)5.24
• Cooperative Multi-agent Reinforcement Learning With Partial Observations (2020)10.35
• Model-based Multi-agent Policy Optimization With Adaptive Opponent-wise Rollouts (2021)0.00
• Offline Multi-agent Reinforcement Learning With Implicit Global-to-local Value Regularization (2023)5.84
• Faster Last-iterate Convergence Of Policy Optimization In Zero-sum Markov Games (2022)0.00
• Representation Learning For Efficient Deep Multi-agent Reinforcement Learning (2024)0.00
• Multi-agent Trust Region Policy Optimization (2020)12.61
• Comadice: Offline Cooperative Multi-agent Reinforcement Learning With Stationary Distribution Shift Regularization (2024)0.00