Uniformly Conservative Exploration In Reinforcement Learning

Abstract

A key challenge to deploying reinforcement learning in practice is avoiding excessive (harmful) exploration in individual episodes. We propose a natural constraint on exploration -- \textit\{uniformly\} outperforming a conservative policy (adaptively estimated from all data observed thus far), up to a per-episode exploration budget. We design a novel algorithm that uses a UCB reinforcement learning policy for exploration, but overrides it as needed to satisfy our exploration constraint with high probability. Importantly, to ensure unbiased exploration across the state space, our algorithm adaptively determines when to explore. We prove that our approach remains conservative while minimizing regret in the tabular setting. We experimentally validate our results on a sepsis treatment task and an HIV treatment task, demonstrating that our algorithm can learn while ensuring good performance compared to the baseline policy for every patient; the latter task also demonstrates that our approac

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxu2021uniformly

Related papers

• Near-optimal Conservative Exploration In Reinforcement Learning Under Episode-wise Constraints (2023)0.00
• Conservative Exploration In Reinforcement Learning (2020)0.00
• Conservative Exploration For Policy Optimization Via Off-policy Policy Evaluation (2023)0.00
• Exploration Conscious Reinforcement Learning Revisited (2018)0.00
• Gap-dependent Unsupervised Exploration For Reinforcement Learning (2021)0.00
• Minimax-optimal Reward-agnostic Exploration In Reinforcement Learning (2023)0.00
• Provably Efficient Exploration For Reinforcement Learning Using Unsupervised Learning (2020)0.00
• Improved Bounds For Reward-agnostic And Reward-free Exploration (2026)0.00