Reward Tampering Problems And Solutions In Reinforcement Learning: A Causal Influence Diagram Perspective

Abstract

Can humans get arbitrarily capable reinforcement learning (RL) agents to do their bidding? Or will sufficiently capable RL agents always find ways to bypass their intended objectives by shortcutting their reward signal? This question impacts how far RL can be scaled, and whether alternative paradigms must be developed in order to build safe artificial general intelligence. In this paper, we study when an RL agent has an instrumental goal to tamper with its reward process, and describe design principles that prevent instrumental goals for two different types of reward tampering (reward function tampering and RF-input tampering). Combined, the design principles can prevent both types of reward tampering from being instrumental goals. The analysis benefits from causal influence diagrams to provide intuitive yet precise formalizations.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyeveritt2019reward

Related papers

• Reward Design For Reinforcement Learning Agents (2025)0.00
• Disturbing Reinforcement Learning Agents With Corrupted Rewards (2021)0.00
• REBEL: Reward Regularization-based Approach For Robotic Reinforcement Learning From Human Feedback (2023)0.00
• The Effects Of Reward Misspecification: Mapping And Mitigating Misaligned Models (2022)0.00
• Causal Campbell-goodhart's Law And Reinforcement Learning (2020)2.26
• Causal Confusion And Reward Misidentification In Preference-based Reward Learning (2022)0.00
• Experiential Explanations For Reinforcement Learning (2022)2.26
• Provably Feedback-efficient Reinforcement Learning Via Active Reward Learning (2023)0.00