Iterative Reward Shaping Using Human Feedback For Correcting Reward Misspecification

Abstract

A well-defined reward function is crucial for successful training of an reinforcement learning (RL) agent. However, defining a suitable reward function is a notoriously challenging task, especially in complex, multi-objective environments. Developers often have to resort to starting with an initial, potentially misspecified reward function, and iteratively adjusting its parameters, based on observed learned behavior. In this work, we aim to automate this process by proposing ITERS, an iterative reward shaping approach using human feedback for mitigating the effects of a misspecified reward function. Our approach allows the user to provide trajectory-level feedback on agent's behavior during training, which can be integrated as a reward shaping signal in the following training iteration. We also allow the user to provide explanations of their feedback, which are used to augment the feedback and reduce user effort and feedback frequency. We evaluate ITERS in three environments and show t

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keygajcin2023iterative

Related papers

• Reward Shaping For Human Learning Via Inverse Reinforcement Learning (2020)0.00
• Provably Feedback-efficient Reinforcement Learning Via Active Reward Learning (2023)0.00
• Highly Efficient Self-adaptive Reward Shaping For Reinforcement Learning (2024)0.00
• ORSO: Accelerating Reward Design Via Online Reward Selection And Policy Optimization (2024)0.00
• Reward Design For Reinforcement Learning Agents (2025)0.00
• SIBRE: Self Improvement Based Rewards For Adaptive Feedback In Reinforcement Learning (2020)0.00
• Automatic Intrinsic Reward Shaping For Exploration In Deep Reinforcement Learning (2023)0.00
• FRESH: Interactive Reward Shaping In High-dimensional State Spaces Using Human Feedback (2020)0.00