Replacing Rewards With Examples: Example-based Policy Search Via Recursive Classification

Abstract

Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another RL algorithm. In contrast, our method directly learns a value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer hyperparameters to tune and lines of code to debug. We show that our method satisfies a new data-driven Bellman equation, where examples take the place of the typical reward function term. Experime

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyeysenbach2021replacing

Related papers

• Reward-conditioned Policies (2019)0.00
• Learning Self-imitating Diverse Policies (2018)0.00
• Improving Policy Gradient By Exploring Under-appreciated Rewards (2016)0.00
• PC-MLP: Model-based Reinforcement Learning With Policy Cover Guided Exploration (2021)0.00
• Reward Design For Reinforcement Learning Agents (2025)0.00
• Discovering Reinforcement Learning Algorithms (2020)0.00
• TGRL: An Algorithm For Teacher Guided Reinforcement Learning (2023)0.00
• Learning Impartial Policies For Sequential Counterfactual Explanations Using Deep Reinforcement Learning (2023)0.00