Near-optimal Conservative Exploration In Reinforcement Learning Under Episode-wise Constraints

Abstract

This paper investigates conservative exploration in reinforcement learning where the performance of the learning agent is guaranteed to be above a certain threshold throughout the learning process. It focuses on the tabular episodic Markov Decision Process (MDP) setting that has finite states and actions. With the knowledge of an existing safe baseline policy, an algorithm termed as StepMix is proposed to balance the exploitation and exploration while ensuring that the conservative constraint is never violated in each episode with high probability. StepMix features a unique design of a mixture policy that adaptively and smoothly interpolates between the baseline policy and the optimistic policy. Theoretical analysis shows that StepMix achieves near-optimal regret order as in the constraint-free setting, indicating that obeying the stringent episode-wise conservative constraint does not compromise the learning performance. Besides, a randomization-based EpsMix algorithm is also proposed

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2023near

Related papers

• Conservative Exploration In Reinforcement Learning (2020)0.00
• Uniformly Conservative Exploration In Reinforcement Learning (2021)0.00
• Improved Bounds For Reward-agnostic And Reward-free Exploration (2026)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Minimax-optimal Reward-agnostic Exploration In Reinforcement Learning (2023)0.00
• Exploration Conscious Reinforcement Learning Revisited (2018)0.00
• An Intrinsically-motivated Approach For Learning Highly Exploring And Fast Mixing Policies (2019)6.34
• Reinforcement Learning In Reward-mixing Mdps (2021)0.00