A Dual Perspective Of Reinforcement Learning For Imposing Policy Constraints

Abstract

Model-free reinforcement learning methods lack an inherent mechanism to impose behavioural constraints on the trained policies. Although certain extensions exist, they remain limited to specific types of constraints, such as value constraints with additional reward signals or visitation density constraints. In this work we unify these existing techniques and bridge the gap with classical optimization and control theory, using a generic primal-dual framework for value-based and actor-critic reinforcement learning methods. The obtained dual formulations turn out to be especially useful for imposing additional constraints on the learned policy, as an intrinsic relationship between such dual constraints (or regularization terms) and reward modifications in the primal is revealed. Furthermore, using this framework, we are able to introduce some novel types of constraints, allowing to impose bounds on the policy's action density or on costs associated with transitions between consecutive sta

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydecooman2024a

Related papers

• State Augmented Constrained Reinforcement Learning: Overcoming The Limitations Of Learning With Rewards (2021)0.00
• Reward Constrained Policy Optimization (2018)0.00
• Handling Cost And Constraints With Off-policy Deep Reinforcement Learning (2023)0.00
• Solving Richly Constrained Reinforcement Learning Through State Augmentation And Reward Penalties (2023)0.00
• Dual RL: Unification And New Methods For Reinforcement And Imitation Learning (2023)0.00
• Concurrent Learning Of Policy And Unknown Safety Constraints In Reinforcement Learning (2024)0.00
• Interpretable Multi-objective Reinforcement Learning Through Policy Orchestration (2018)0.00
• Achieving Zero Constraint Violation For Constrained Reinforcement Learning Via Primal-dual Approach (2021)9.59