Exterior Penalty Policy Optimization With Penalty Metric Network Under Constraints

Abstract

In Constrained Reinforcement Learning (CRL), agents explore the environment to learn the optimal policy while satisfying constraints. The penalty function method has recently been studied as an effective approach for handling constraints, which imposes constraints penalties on the objective to transform the constrained problem into an unconstrained one. However, it is challenging to choose appropriate penalties that balance policy performance and constraint satisfaction efficiently. In this paper, we propose a theoretically guaranteed penalty function method, Exterior Penalty Policy Optimization (EPO), with adaptive penalties generated by a Penalty Metric Network (PMN). PMN responds appropriately to varying degrees of constraint violations, enabling efficient constraint satisfaction and safe exploration. We theoretically prove that EPO consistently improves constraint satisfaction with a convergence guarantee. We propose a new surrogate function and provide worst-case constraint violat

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keygao2024exterior

Related papers

• Reward Constrained Policy Optimization (2018)0.00
• Solving Richly Constrained Reinforcement Learning Through State Augmentation And Reward Penalties (2023)0.00
• CRPO: A New Approach For Safe Reinforcement Learning With Convergence Guarantee (2020)0.00
• Redeeming Intrinsic Rewards Via Constrained Optimization (2022)0.00
• Co2po: Coordinated Constrained Policy Optimization For Multi-agent RL (2026)0.00
• Efficient Policy Optimization In Robust Constrained Mdps With Iteration Complexity Guarantees (2025)0.00
• Conservative And Adaptive Penalty For Model-based Safe Reinforcement Learning (2021)0.00
• Learning Deterministic Policies With Policy Gradients In Constrained Markov Decision Processes (2025)0.00