REVEAL-IT: Reinforcement Learning With Visibility Of Evolving Agent Policy For Interpretability

Abstract

Understanding the agent's learning process, particularly the factors that contribute to its success or failure post-training, is crucial for comprehending the rationale behind the agent's decision-making process. Prior methods clarify the learning process by creating a structural causal model (SCM) or visually representing the distribution of value functions. Nevertheless, these approaches have constraints as they exclusively function in 2D-environments or with uncomplicated transition dynamics. Understanding the agent's learning process in complicated environments or tasks is more challenging. In this paper, we propose REVEAL-IT, a novel framework for explaining the learning process of an agent in complex environments. Initially, we visualize the policy structure and the agent's learning process for various training tasks. By visualizing these findings, we can understand how much a particular training task or stage affects the agent's performance in test. Then, a GNN-based explainer l

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyao2024reveal

Related papers

• Why The Agent Made That Decision: Contrastive Explanation Learning For Reinforcement Learning (2024)0.00
• "so, Tell Me About Your Policy...": Distillation Of Interpretable Policies From Deep Reinforcement Learning Agents (2025)0.00
• Interpretable Learning Dynamics In Unsupervised Reinforcement Learning (2025)0.00
• REACT: Revealing Evolutionary Action Consequence Trajectories For Interpretable Reinforcement Learning (2024)2.26
• How Do You Act? An Empirical Study To Understand Behavior Of Deep Reinforcement Learning Agents (2020)0.00
• A Framework For Understanding And Visualizing Strategies Of RL Agents (2022)0.00
• From Explainability To Interpretability: Interpretable Policies In Reinforcement Learning Via Model Explanation (2025)0.00
• Integrating Policy Summaries With Reward Decomposition For Explaining Reinforcement Learning Agents (2022)7.16