Modeling Strong And Human-like Gameplay With Kl-regularized Search

Abstract

We consider the task of building strong but human-like policies in multi-agent decision-making problems, given examples of human behavior. Imitation learning is effective at predicting human actions but may not match the strength of expert humans, while self-play learning and search techniques (e.g. AlphaZero) lead to strong performance but may produce policies that are difficult for humans to understand and coordinate with. We show in chess and Go that regularizing search based on the KL divergence from an imitation-learned policy results in higher human prediction accuracy and stronger performance than imitation learning alone. We then introduce a novel regret minimization algorithm that is regularized based on the KL divergence from an imitation-learned policy, and show that using this algorithm for search in no-press Diplomacy yields a policy that matches the human prediction accuracy of imitation learning while being substantially stronger.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyjacob2021modeling

Related papers

• Human-ai Coordination Via Human-regularized Search And Learning (2022)0.00
• Mastering The Game Of No-press Diplomacy Via Human-regularized Reinforcement Learning And Planning (2022)0.00
• Policy-value Alignment And Robustness In Search-based Multi-agent Learning (2023)0.00
• Learning To Play No-press Diplomacy With Best Response Policy Iteration (2020)0.00
• Regret-guided Search Control For Efficient Learning In Alphazero (2026)0.00
• Impartial Games: A Challenge For Reinforcement Learning (2022)0.00
• Mimicking To Dominate: Imitation Learning Strategies For Success In Multiagent Competitive Games (2023)0.00
• Pessimism-free Offline Learning In General-sum Games Via KL Regularization (2026)0.00