Unpacking Reward Shaping: Understanding The Benefits Of Reward Engineering On Sample Complexity

Abstract

Reinforcement learning provides an automated framework for learning behaviors from high-level reward specifications, but in practice the choice of reward function can be crucial for good results -- while in principle the reward only needs to specify what the task is, in reality practitioners often need to design more detailed rewards that provide the agent with some hints about how the task should be completed. The idea of this type of ``reward-shaping'' has been often discussed in the literature, and is often a critical part of practical applications, but there is relatively little formal characterization of how the choice of reward shaping can yield benefits in sample complexity. In this work, we build on the framework of novelty-based exploration to provide a simple scheme for incorporating shaped rewards into RL along with an analysis tool to show that particular choices of reward shaping provably improve sample efficiency. We characterize the class of problems where these gains ar

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keygupta2022unpacking

Related papers

• Highly Efficient Self-adaptive Reward Shaping For Reinforcement Learning (2024)0.00
• Reward Design For Reinforcement Learning Agents (2025)0.00
• BAMDP Shaping: A Unified Framework For Intrinsic Motivation And Reward Shaping (2024)0.00
• Multimodal Reward Shaping For Efficient Exploration In Reinforcement Learning (2021)0.00
• ORSO: Accelerating Reward Design Via Online Reward Selection And Policy Optimization (2024)0.00
• Optimistic Curiosity Exploration And Conservative Exploitation With Linear Reward Shaping (2022)0.00
• Learning To Shape Rewards Using A Game Of Two Partners (2021)0.00
• Action Guidance: Getting The Best Of Sparse Rewards And Shaped Rewards For Real-time Strategy Games (2020)0.00