Promoting Coordination Through Policy Regularization In Multi-agent Deep Reinforcement Learning

Abstract

In multi-agent reinforcement learning, discovering successful collective behaviors is challenging as it requires exploring a joint action space that grows exponentially with the number of agents. While the tractability of independent agent-wise exploration is appealing, this approach fails on tasks that require elaborate group strategies. We argue that coordinating the agents' policies can guide their exploration and we investigate techniques to promote such an inductive bias. We propose two policy regularization methods: TeamReg, which is based on inter-agent action predictability and CoachReg that relies on synchronized behavior selection. We evaluate each approach on four challenging continuous control tasks with sparse rewards that require varying levels of coordination as well as on the discrete action Google Research Football environment. Our experiments show improved performance across many cooperative multi-agent problems. Finally, we analyze the effects of our proposed methods

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyroy2019promoting

Related papers

• Regularize! Don't Mix: Multi-agent Reinforcement Learning Without Explicit Centralized Structures (2021)0.00
• Coordinated Exploration Via Intrinsic Rewards For Multi-agent Reinforcement Learning (2019)0.00
• GCS: Graph-based Coordination Strategy For Multi-agent Reinforcement Learning (2022)0.00
• Collaboration Of AI Agents Via Cooperative Multi-agent Deep Reinforcement Learning (2019)0.00
• Influence-based Reinforcement Learning For Intrinsically-motivated Agents (2021)0.00
• Signal Instructed Coordination In Cooperative Multi-agent Reinforcement Learning (2019)4.52
• Regularization Matters In Policy Optimization (2019)2.68
• Multi-agent Actor-critic For Mixed Cooperative-competitive Environments (2017)0.00