Objective Metrics For Human-subjects Evaluation In Explainable Reinforcement Learning

Abstract

Explanation is a fundamentally human process. Understanding the goal and audience of the explanation is vital, yet existing work on explainable reinforcement learning (XRL) routinely does not consult humans in their evaluations. Even when they do, they routinely resort to subjective metrics, such as confidence or understanding, that can only inform researchers of users' opinions, not their practical effectiveness for a given problem. This paper calls on researchers to use objective human metrics for explanation evaluations based on observable and actionable behaviour to build more reproducible, comparable, and epistemically grounded research. To this end, we curate, describe, and compare several objective evaluation methodologies for applying explanations to debugging agent behaviour and supporting human-agent teaming, illustrating our proposed methods using a novel grid-based environment. We discuss how subjective and objective metrics complement each other to provide holistic validat

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygyevnar2025objective

Related papers

• Xrl-bench: A Benchmark For Evaluating And Comparing Explainable Reinforcement Learning Techniques (2024)0.00
• A Survey Of Explainable Reinforcement Learning: Targets, Methods And Needs (2025)0.00
• A Survey Of Explainable Reinforcement Learning (2022)0.00
• Interestingness Elements For Explainable Reinforcement Learning: Understanding Agents' Capabilities And Limitations (2019)14.55
• A Comparative User Evaluation Of XRL Explanations Using Goal Identification (2025)0.00
• A Survey On Explainable Reinforcement Learning: Concepts, Algorithms, Challenges (2022)0.00
• Experiential Explanations For Reinforcement Learning (2022)2.26
• Explainability In Deep Reinforcement Learning (2020)0.00