Learning To Play No-press Diplomacy With Best Response Policy Iteration

Abstract

Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually simple and principled application of RL methods. However real-world settings are many-agent, and agent interactions are complex mixtures of common-interest and competitive aspects. We consider Diplomacy, a 7-player board game designed to accentuate dilemmas resulting from many-agent interactions. It also features a large combinatorial action space and simultaneous moves, which are challenging for RL algorithms. We propose a simple yet effective approximate best response operator, designed to handle large combinatorial action spaces and simultaneous moves. We also introduce a family of policy iteration methods that approximate fictitious play. With these methods, we successfully apply RL to Diplomacy: we show that our agents convincingly outperform the previous state-of-

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyanthony2020learning

Related papers

• Mastering The Game Of No-press Diplomacy Via Human-regularized Reinforcement Learning And Planning (2022)0.00
• Modeling Strong And Human-like Gameplay With Kl-regularized Search (2021)0.00
• Simplified Action Decoder For Deep Multi-agent Reinforcement Learning (2019)4.03
• Adversarial Policies: Attacking Deep Reinforcement Learning (2019)0.00
• Brexit: On Opponent Modelling In Expert Iteration (2022)0.00
• Mastering Strategy Card Game (legends Of Code And Magic) Via End-to-end Policy And Optimistic Smooth Fictitious Play (2023)0.00
• Efficient Competitive Self-play Policy Optimization (2020)0.00
• Reevaluating Policy Gradient Methods For Imperfect-information Games (2025)0.00