Safe Offline Reinforcement Learning With Real-time Budget Constraints

Abstract

Aiming at promoting the safe real-world deployment of Reinforcement Learning (RL), research on safe RL has made significant progress in recent years. However, most existing works in the literature still focus on the online setting where risky violations of the safety budget are likely to be incurred during training. Besides, in many real-world applications, the learned policy is required to respond to dynamically determined safety budgets (i.e., constraint threshold) in real time. In this paper, we target at the above real-time budget constraint problem under the offline setting, and propose Trajectory-based REal-time Budget Inference (TREBI) as a novel solution that models this problem from the perspective of trajectory distribution and solves it through diffusion model planning. Theoretically, we prove an error bound of the estimation on the episodic reward and cost under the offline setting and thus provide a performance guarantee for TREBI. Empirical results on a wide range of simu

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylin2023safe

Related papers

• Offline Safe Reinforcement Learning Using Trajectory Classification (2024)0.00
• Constraints Penalized Q-learning For Safe Offline Reinforcement Learning (2021)0.00
• One Risk To Rule Them All: A Risk-sensitive Perspective On Model-based Offline Reinforcement Learning (2022)3.58
• Towards Fast Safe Online Reinforcement Learning Via Policy Finetuning (2024)0.00
• Provably Efficient Offline Reinforcement Learning With Trajectory-wise Reward (2022)0.00
• Decoupled Guidance Diffusion For Adaptive Offline Safe Reinforcement Learning (2026)0.00
• Optimal Transport Perturbations For Safe Reinforcement Learning With Robustness Guarantees (2023)0.00
• Context-aware Safe Reinforcement Learning For Non-stationary Environments (2021)9.76