Safe-support Q-learning: Learning Without Unsafe Exploration

Abstract

arXiv:2604.25379v1 Announce Type: new Abstract: Ensuring safety during reinforcement learning (RL) training is critical in real-world applications where unsafe exploration can lead to devastating outcomes. While most safe RL methods mitigate risk through constraints or penalization, they still allow exploration of unsafe states during training. In this work, we adopt a stricter safety requirement that eliminates unsafe state visitation during training. To achieve this goal, we propose a Q-learning-based safe RL framework that leverages a behavior policy supported on a safe set. Under the assumption that the induced trajectories remain within the safe set, this policy enables sufficient exploration within the safe region without requiring near-optimality. We adopt a two-stage framework in which the Q-function and policy are trained separately. Specifically, we introduce a KL-regularized Bellman target that constrains the Q-function to remain close to the behavior policy. We then derive

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylim2026safe

Related papers

• Actsafe: Active Exploration With Safety Constraints For Reinforcement Learning (2024)0.00
• Reinforcement Learning By Guided Safe Exploration (2023)5.24
• Safe Reinforcement Learning Via Projection On A Safe Set: How To Achieve Optimality? (2020)0.00
• Safe Policy Optimization With Local Generalized Linear Function Approximations (2021)0.00
• Constraints Penalized Q-learning For Safe Offline Reinforcement Learning (2021)0.00
• Safety Modulation: Enhancing Safety In Reinforcement Learning Through Cost-modulated Rewards (2025)0.00
• Safe Reinforcement Learning With Dual Robustness (2023)8.60
• Implicit Safe Set Algorithm For Provably Safe Reinforcement Learning (2024)0.00