Pitfalls Of Learning A Reward Function Online

Abstract

In some agent designs like inverse reinforcement learning an agent needs to learn its own reward function. Learning the reward function and optimising for it are typically two different processes, usually performed at different stages. We consider a continual (``one life'') learning approach where the agent both learns the reward function and optimises for it at the same time. We show that this comes with a number of pitfalls, such as deliberately manipulating the learning process in one direction, refusing to learn, ``learning'' facts already known to the agent, and making decisions that are strictly dominated (for all relevant reward functions). We formally introduce two desirable properties: the first is `unriggability', which prevents the agent from steering the learning process in the direction of a reward function that is easier to optimise. The second is `uninfluenceability', whereby the reward-function learning process operates by learning facts about the environment. We show t

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyarmstrong2020pitfalls

Related papers

• Designing Rewards For Fast Learning (2022)0.00
• Reward Design For Reinforcement Learning Agents (2025)0.00
• Informativeness Of Reward Functions In Reinforcement Learning (2024)2.26
• Goodhart's Law In Reinforcement Learning (2023)0.00
• An Agent Design With Goal Reaching Guarantees For Enhancement Of Learning (2024)0.00
• Invariance In Policy Optimisation And Partial Identifiability In Reward Learning (2022)0.00
• What Can Learned Intrinsic Rewards Capture? (2019)0.00
• On Learning Intrinsic Rewards For Policy Gradient Methods (2018)0.00