Tell Me Why: Training Preferences-based RL With Human Preferences And Step-level Explanations

Abstract

Human-in-the-loop reinforcement learning allows the training of agents through various interfaces, even for non-expert humans. Recently, preference-based methods (PbRL), where the human has to give his preference over two trajectories, increased in popularity since they allow training in domains where more direct feedback is hard to formulate. However, the current PBRL methods have limitations and do not provide humans with an expressive interface for giving feedback. With this work, we propose a new preference-based learning method that provides humans with a more expressive interface to provide their preference over trajectories and a factual explanation (or annotation of why they have this preference). These explanations allow the human to explain what parts of the trajectory are most relevant for the preference. We allow the expression of the explanations over individual trajectory steps. We evaluate our method in various simulations using a simulated human oracle (with realistic r

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykaralus2024tell

Related papers

• Dueling RL: Reinforcement Learning With Trajectory Preferences (2021)0.00
• Reinforcement Learning From Diverse Human Preferences (2023)0.00
• Symbol Guided Hindsight Priors For Reward Learning From Human Preferences (2022)0.00
• Hindsight Priors For Reward Learning From Human Preferences (2024)0.00
• Experiential Explanations For Reinforcement Learning (2022)2.26
• Pref-guide: Continual Policy Learning From Real-time Human Feedback Via Preference-based Learning (2025)0.00
• Data Driven Reward Initialization For Preference Based Reinforcement Learning (2023)0.00
• Ra-pbrl: Provably Efficient Risk-aware Preference-based Reinforcement Learning (2024)0.00