Evaluation Of Human-ai Teams For Learned And Rule-based Agents In Hanabi

Abstract

Deep reinforcement learning has generated superhuman AI in competitive games such as Go and StarCraft. Can similar learning techniques create a superior AI teammate for human-machine collaborative games? Will humans prefer AI teammates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform a single-blind evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule-based and learning-based agents. In addition to the game score, used as an objective metric of the human-AI team performance, we also quantify subjective measures of the human's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have a clear preference toward a rule-based AI teammate (SmartBot) over a state-of-the-art learning-based AI teammate (Other-Play) across nearly all subjective metrics, and generally view the learning-based agent negatively, despite no stati

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysiu2021evaluation

Related papers

• In Pursuit Of Predictive Models Of Human Preferences Toward AI Teammates (2025)0.00
• Reinforcement Learning On Human Decision Models For Uniquely Collaborative AI Teammates (2021)0.00
• Evaluating The Rainbow DQN Agent In Hanabi With Unseen Partners (2020)0.00
• Simplified Action Decoder For Deep Multi-agent Reinforcement Learning (2019)4.03
• Human-ai Coordination Via Human-regularized Search And Learning (2022)0.00
• Theory Of Mind For Deep Reinforcement Learning In Hanabi (2021)0.00
• Behavioral Differences Is The Key Of Ad-hoc Team Cooperation In Multiplayer Games Hanabi (2023)0.00
• Winning Isn't Everything: Enhancing Game Development With Intelligent Agents (2019)11.29