DOPE: Doubly Optimistic And Pessimistic Exploration For Safe Reinforcement Learning

Abstract

Safe reinforcement learning is extremely challenging--not only must the agent explore an unknown environment, it must do so while ensuring no safety constraint violations. We formulate this safe reinforcement learning (RL) problem using the framework of a finite-horizon Constrained Markov Decision Process (CMDP) with an unknown transition probability function, where we model the safety requirements as constraints on the expected cumulative costs that must be satisfied during all episodes of learning. We propose a model-based safe RL algorithm that we call Doubly Optimistic and Pessimistic Exploration (DOPE), and show that it achieves an objective regret \(\tilde\{O\}(|\mathcal\{S\}|\sqrt\{|\mathcal\{A\}| K\})\) without violating the safety constraints during learning, where \(|\mathcal\{S\}|\) is the number of states, \(|\mathcal\{A\}|\) is the number of actions, and \(K\) is the number of learning episodes. Our key idea is to combine a reward bonus for exploration (optimism) with a co

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybura2021dope

Related papers

• Model-based Safe Deep Reinforcement Learning Via A Constrained Proximal Policy Optimization Algorithm (2022)5.24
• Actsafe: Active Exploration With Safety Constraints For Reinforcement Learning (2024)0.00
• Provably Efficient Primal-dual Reinforcement Learning For Cmdps With Non-stationary Objectives And Constraints (2022)0.00
• Conservative And Adaptive Penalty For Model-based Safe Reinforcement Learning (2021)0.00
• Safety Modulation: Enhancing Safety In Reinforcement Learning Through Cost-modulated Rewards (2025)0.00
• Safe Reinforcement Learning With Dual Robustness (2023)8.60
• Enhancing Efficiency Of Safe Reinforcement Learning Via Sample Manipulation (2024)0.00
• Implicit Safe Set Algorithm For Provably Safe Reinforcement Learning (2024)0.00