Reward Constrained Policy Optimization

Abstract

Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytessler2018reward

Related papers

• CRPO: A New Approach For Safe Reinforcement Learning With Convergence Guarantee (2020)0.00
• Constraint-conditioned Policy Optimization For Versatile Safe Reinforcement Learning (2023)0.00
• Exterior Penalty Policy Optimization With Penalty Metric Network Under Constraints (2024)0.00
• Continuously Discovering Novel Strategies Via Reward-switching Policy Optimization (2022)0.00
• Solving Richly Constrained Reinforcement Learning Through State Augmentation And Reward Penalties (2023)0.00
• Think Outside The Policy: In-context Steered Policy Optimization (2025)0.00
• Model-based Safe Deep Reinforcement Learning Via A Constrained Proximal Policy Optimization Algorithm (2022)5.24
• Co2po: Coordinated Constrained Policy Optimization For Multi-agent RL (2026)0.00