Brexit: On Opponent Modelling In Expert Iteration

Abstract

Finding a best response policy is a central objective in game theory and multi-agent learning, with modern population-based training approaches employing reinforcement learning algorithms as best-response oracles to improve play against candidate opponents (typically previously learnt policies). We propose Best Response Expert Iteration (BRExIt), which accelerates learning in games by incorporating opponent models into the state-of-the-art learning algorithm Expert Iteration (ExIt). BRExIt aims to (1) improve feature shaping in the apprentice, with a policy head predicting opponent policies as an auxiliary task, and (2) bias opponent moves in planning towards the given or learnt opponent model, to generate apprentice targets that better approximate a best response. In an empirical ablation on BRExIt's algorithmic variants against a set of fixed test agents, we provide statistical evidence that BRExIt learns better performing policies than ExIt.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhernandez2022brexit

Related papers

• Combining Tree-search, Generative Models, And Nash Bargaining Concepts In Game-theoretic Reinforcement Learning (2023)0.00
• Learning To Play No-press Diplomacy With Best Response Policy Iteration (2020)0.00
• Manipulating The Distributions Of Experience Used For Self-play Learning In Expert Iteration (2020)5.24
• Approximate Exploitability: Learning A Best Response In Large Games (2020)0.00
• Learning Equilibria From Data: Provably Efficient Multi-agent Imitation Learning (2025)0.00
• Model-based Opponent Modeling (2021)0.00
• Consistent Opponent Modeling In Imperfect-information Games (2025)0.00
• Online Matching Via Reinforcement Learning: An Expert Policy Orchestration Strategy (2025)0.00