Constraint-conditioned Policy Optimization For Versatile Safe Reinforcement Learning

Abstract

Safe reinforcement learning (RL) focuses on training reward-maximizing agents subject to pre-defined safety constraints. Yet, learning versatile safe policies that can adapt to varying safety constraint requirements during deployment without retraining remains a largely unexplored and challenging area. In this work, we formulate the versatile safe RL problem and consider two primary requirements: training efficiency and zero-shot adaptation capability. To address them, we introduce the Conditioned Constrained Policy Optimization (CCPO) framework, consisting of two key modules: (1) Versatile Value Estimation (VVE) for approximating value functions under unseen threshold conditions, and (2) Conditioned Variational Inference (CVI) for encoding arbitrary constraint thresholds during policy optimization. Our extensive experiments demonstrate that CCPO outperforms the baselines in terms of safety and task performance while preserving zero-shot adaptation capabilities to different constraint

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyao2023constraint

Related papers

• Reward Constrained Policy Optimization (2018)0.00
• Towards Safe Reinforcement Learning Via Constraining Conditional Value-at-risk (2022)0.00
• Concurrent Learning Of Policy And Unknown Safety Constraints In Reinforcement Learning (2024)0.00
• Model-based Safe Deep Reinforcement Learning Via A Constrained Proximal Policy Optimization Algorithm (2022)5.24
• Safety Modulation: Enhancing Safety In Reinforcement Learning Through Cost-modulated Rewards (2025)0.00
• Embedding Safety Into RL: A New Take On Trust Region Methods (2024)0.00
• CRPO: A New Approach For Safe Reinforcement Learning With Convergence Guarantee (2020)0.00
• Multi-agent Constrained Policy Optimisation (2021)0.00