Explainable Reinforcement Learning Through A Causal Lens

Abstract

Prevalent theories in cognitive science propose that humans understand and represent the knowledge of the world through causal relationships. In making sense of the world, we build causal models in our mind to encode cause-effect relations of events and use these to explain why new events happen. In this paper, we use causal models to derive causal explanations of behaviour of reinforcement learning agents. We present an approach that learns a structural causal model during reinforcement learning and encodes causal relationships between variables of interest. This model is then used to generate explanations of behaviour based on counterfactual analysis of the causal model. We report on a study with 120 participants who observe agents playing a real-time strategy game (Starcraft II) and then receive explanations of the agents' behaviour. We investigated: 1) participants' understanding gained by explanations through task prediction; 2) explanation satisfaction and 3) trust. Our results s

Stats

• citations167
• S2 citations—
• github stars0
• HF likes0
• heat score16.69
• arxiv keymadumal2019explainable

Related papers

• Explainable Reinforcement Learning Via A Causal World Model (2023)9.03
• Learning By Doing: An Online Causal Reinforcement Learning Framework With Causal-aware Policy (2024)1.56
• Experiential Explanations For Reinforcement Learning (2022)2.26
• Why Online Reinforcement Learning Is Causal (2024)0.00
• Learning Nonlinear Causal Reductions To Explain Reinforcement Learning Policies (2025)0.00
• Causal State Distillation For Explainable Reinforcement Learning (2023)0.00
• Causal Reinforcement Learning: A Survey (2023)0.00
• Reccover: Detecting Causal Confusion For Explainable Reinforcement Learning (2022)0.00