Solving Richly Constrained Reinforcement Learning Through State Augmentation And Reward Penalties

Abstract

Constrained Reinforcement Learning has been employed to enforce safety constraints on policy through the use of expected cost constraints. The key challenge is in handling expected cost accumulated using the policy and not just in a single step. Existing methods have developed innovative ways of converting this cost constraint over entire policy to constraints over local decisions (at each time step). While such approaches have provided good solutions with regards to objective, they can either be overly aggressive or conservative with respect to costs. This is owing to use of estimates for "future" or "backward" costs in local cost constraints. To that end, we provide an equivalent unconstrained formulation to constrained RL that has an augmented state space and reward penalties. This intuitive formulation is general and has interesting theoretical properties. More importantly, this provides a new paradigm for solving constrained RL problems effectively. As we show in our experimenta

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyjiang2023solving

Related papers

• State Augmented Constrained Reinforcement Learning: Overcoming The Limitations Of Learning With Rewards (2021)0.00
• Reward Constrained Policy Optimization (2018)0.00
• Handling Cost And Constraints With Off-policy Deep Reinforcement Learning (2023)0.00
• Model-based Safe Deep Reinforcement Learning Via A Constrained Proximal Policy Optimization Algorithm (2022)5.24
• Reinforcement Learning With Convex Constraints (2019)0.00
• Concurrent Learning Of Policy And Unknown Safety Constraints In Reinforcement Learning (2024)0.00
• Conservative And Adaptive Penalty For Model-based Safe Reinforcement Learning (2021)0.00
• Imitate The Good And Avoid The Bad: An Incremental Approach To Safe Reinforcement Learning (2023)0.00