The Perils Of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret

Abstract

In reinforcement learning, specifying reward functions that capture the intended task can be very challenging. Reward learning aims to address this issue by learning the reward function. However, a learned reward model may have a low error on the data distribution, and yet subsequently produce a policy with large regret. We say that such a reward model has an error-regret mismatch. The main source of an error-regret mismatch is the distributional shift that commonly occurs during policy optimization. In this paper, we mathematically show that a sufficiently low expected test error of the reward model guarantees low worst-case regret, but that for any fixed expected test error, there exist realistic data distributions that allow for error-regret mismatch to occur. We then show that similar problems persist even when using policy regularization techniques, commonly employed in methods such as RLHF. We hope our results stimulate the theoretical and empirical study of improved methods to l

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfluri2024the

Related papers

• When Errors Can Be Beneficial: A Categorization Of Imperfect Rewards For Policy Gradient (2026)0.00
• Provably Mitigating Overoptimization In RLHF: Your SFT Loss Is Implicitly An Adversarial Regularizer (2024)0.00
• Goodhart's Law In Reinforcement Learning (2023)0.00
• Wasserstein Distributionally Robust Regret Optimization For Reinforcement Learning From Human Feedback (2026)0.00
• Pitfalls Of Learning A Reward Function Online (2020)4.52
• The Best Of Both Worlds: Reinforcement Learning With Logarithmic Regret And Policy Switches (2022)0.00
• Expected Return Causes Outcome-level Mode Collapse In Reinforcement Learning And How To Fix It With Inverse Probability Scaling (2026)0.00
• Learning The Reward Function For A Misspecified Model (2018)0.00