Causal Confusion And Reward Misidentification In Preference-based Reward Learning

Abstract

Learning policies via preference-based reward learning is an increasingly popular method for customizing agent behavior, but has been shown anecdotally to be prone to spurious correlations and reward hacking behaviors. While much prior work focuses on causal confusion in reinforcement learning and behavioral cloning, we focus on a systematic study of causal confusion and reward misidentification when learning from preferences. In particular, we perform a series of sensitivity and ablation analyses on several benchmark domains where rewards learned from preferences achieve minimal test error but fail to generalize to out-of-distribution states -- resulting in poor policy performance when optimized. We find that the presence of non-causal distractor features, noise in the stated preferences, and partial state observability can all exacerbate reward misidentification. We also identify a set of methods with which to interpret misidentified learned rewards. In general, we observe that optim

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytien2022causal

Related papers

• The Effects Of Reward Misspecification: Mapping And Mitigating Misaligned Models (2022)0.00
• Breaking Habits: On The Role Of The Advantage Function In Learning Causal State Representations (2025)0.00
• Invariance In Policy Optimisation And Partial Identifiability In Reward Learning (2022)0.00
• Reward-conditioned Policies (2019)0.00
• Reward Tampering Problems And Solutions In Reinforcement Learning: A Causal Influence Diagram Perspective (2019)0.00
• Correlated Proxies: A New Definition And Improved Mitigation For Reward Hacking (2024)2.76
• Reward Model Learning Vs. Direct Policy Optimization: A Comparative Analysis Of Learning From Human Preferences (2024)0.00
• Learning Nonlinear Causal Reductions To Explain Reinforcement Learning Policies (2025)0.00