Can RLHF Be More Efficient With Imperfect Reward Models? A Policy Coverage Perspective

Abstract

Sample efficiency is critical for online Reinforcement Learning from Human Feedback (RLHF). While existing works investigate sample-efficient online exploration strategies, the potential of utilizing misspecified yet relevant reward models to accelerate learning remains underexplored. This paper studies how to transfer knowledge from those imperfect reward models in online RLHF. We start by identifying a novel property due to KL-regularization in the RLHF objective: *a policy's coverability of the optimal policy is captured by its sub-optimality*. Building on this insight, we propose novel transfer learning principles and a theoretical algorithm -- *\textbf\{T*ransfer \textbf\{P\}olicy \textbf\{O\}ptimization (\textbf\{TPO\})\} -- with provable benefits compared to standard online learning. Empirically, inspired by our theoretical findings, we develop a win-rate-based transfer policy selection strategy with improved computational efficiency. Moreover, our empirical transfer learning te

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhuang2025can

Related papers

• Exploration-driven Policy Optimization In RLHF: Theoretical Insights On Efficient Data Utilization (2024)0.00
• Policy Filtration For RLHF To Mitigate Noise In Reward Models (2024)0.00
• Provably Mitigating Overoptimization In RLHF: Your SFT Loss Is Implicitly An Adversarial Regularizer (2024)0.00
• Reward-agnostic Fine-tuning: Provable Statistical Benefits Of Hybrid Reinforcement Learning (2023)0.00
• An Advantage Based Policy Transfer Algorithm For Reinforcement Learning With Measures Of Transferability (2023)0.00
• Asynchronous RLHF: Faster And More Efficient Off-policy RL For Language Models (2024)0.00
• Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency From Shifted-dynamics Data (2024)0.00
• Understanding The Performance Gap In Preference Learning: A Dichotomy Of RLHF And DPO (2025)0.00