Maximum Entropy Population-based Training For Zero-shot Human-ai Coordination

Abstract

We study the problem of training a Reinforcement Learning (RL) agent that is collaborative with humans without using any human data. Although such agents can be obtained through self-play training, they can suffer significantly from distributional shift when paired with unencountered partners, such as humans. To mitigate this distributional shift, we propose Maximum Entropy Population-based training (MEP). In MEP, agents in the population are trained with our derived Population Entropy bonus to promote both pairwise diversity between agents and individual diversity of agents themselves, and a common best agent is trained by paring with agents in this diversified population via prioritized sampling. The prioritization is dynamically adjusted based on the training progress. We demonstrate the effectiveness of our method MEP, with comparison to Self-Play PPO (SP), Population-Based Training (PBT), Trajectory Diversity (TrajeDi), and Fictitious Co-Play (FCP) in the Overcooked game environme

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhao2021maximum

Related papers

• Learning Zero-shot Cooperation With Humans, Assuming Humans Are Biased (2023)0.00
• Collaborating With Humans Without Human Data (2021)0.00
• A Hierarchical Approach To Population Training For Human-ai Collaboration (2023)0.00
• Tackling Cooperative Incompatibility For Zero-shot Human-ai Coordination (2023)0.00
• Heterogeneous Multi-agent Zero-shot Coordination By Coevolution (2022)5.24
• PECAN: Leveraging Policy Ensemble For Context-aware Zero-shot Human-ai Coordination (2023)2.26
• Zero Shot Coordination For Sparse Reward Tasks With Diverse Reward Shapings (2026)0.00
• "other-play" For Zero-shot Coordination (2020)0.00