Data Driven Reward Initialization For Preference Based Reinforcement Learning

Abstract

Preference-based Reinforcement Learning (PbRL) methods utilize binary feedback from the human in the loop (HiL) over queried trajectory pairs to learn a reward model in an attempt to approximate the human's underlying reward function capturing their preferences. In this work, we investigate the issue of a high degree of variability in the initialized reward models which are sensitive to random seeds of the experiment. This further compounds the issue of degenerate reward functions PbRL methods already suffer from. We propose a data-driven reward initialization method that does not add any additional cost to the human in the loop and negligible cost to the PbRL agent and show that doing so ensures that the predicted rewards of the initialized reward model are uniform in the state space and this reduces the variability in the performance of the method across multiple runs and is shown to improve the overall performance compared to other initialization methods.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyverma2023data

Related papers

• Hindsight Priors For Reward Learning From Human Preferences (2024)0.00
• Symbol Guided Hindsight Priors For Reward Learning From Human Preferences (2022)0.00
• Efficient Preference-based Reinforcement Learning Via Aligned Experience Estimation (2024)0.00
• Dueling RL: Reinforcement Learning With Trajectory Preferences (2021)0.00
• Listwise Reward Estimation For Offline Preference-based Reinforcement Learning (2024)0.00
• Ra-pbrl: Provably Efficient Risk-aware Preference-based Reinforcement Learning (2024)0.00
• Evaluating Feature Dependent Noise In Preference-based Reinforcement Learning (2026)0.00
• Reinforcement Learning From Diverse Human Preferences (2023)0.00