Explaining Learned Reward Functions With Counterfactual Trajectories

Abstract

Learning rewards from human behaviour or feedback is a promising approach to aligning AI systems with human values but fails to consistently extract correct reward functions. Interpretability tools could enable users to understand and evaluate possible flaws in learned reward functions. We propose Counterfactual Trajectory Explanations (CTEs) to interpret reward functions in reinforcement learning by contrasting an original with a counterfactual partial trajectory and the rewards they each receive. We derive six quality criteria for CTEs and propose a novel Monte-Carlo-based algorithm for generating CTEs that optimises these quality criteria. Finally, we measure how informative the generated explanations are to a proxy-human model by training it on CTEs. CTEs are demonstrably informative for the proxy-human model, increasing the similarity between its predictions and the reward function on unseen trajectories. Further, it learns to accurately judge differences in rewards between trajec

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywehner2024explaining

Related papers

• Explaining Reinforcement Learning Policies Through Counterfactual Trajectories (2022)0.00
• Redefining Counterfactual Explanations For Reinforcement Learning: Overview, Challenges And Opportunities (2022)0.00
• RACCER: Towards Reachable And Certain Counterfactual Explanations For Reinforcement Learning (2023)0.00
• Counterfactual State Explanations For Reinforcement Learning Agents Via Generative Deep Learning (2021)13.23
• Learning "what-if" Explanations For Sequential Decision-making (2020)0.00
• ACTER: Diverse And Actionable Counterfactual Sequences For Explaining And Diagnosing RL Policies (2024)0.00
• Learning Impartial Policies For Sequential Counterfactual Explanations Using Deep Reinforcement Learning (2023)0.00
• Experiential Explanations For Reinforcement Learning (2022)2.26