Uncertainty Quantification And Exploration For Reinforcement Learning

Abstract

We investigate statistical uncertainty quantification for reinforcement learning (RL) and its implications in exploration policy. Despite ever-growing literature on RL applications, fundamental questions about inference and error quantification, such as large-sample behaviors, appear to remain quite open. In this paper, we fill in the literature gap by studying the central limit theorem behaviors of estimated Q-values and value functions under various RL settings. In particular, we explicitly identify closed-form expressions of the asymptotic variances, which allow us to efficiently construct asymptotically valid confidence regions for key RL quantities. Furthermore, we utilize these asymptotic expressions to design an effective exploration strategy, which we call Q-value-based Optimal Computing Budget Allocation (Q-OCBA). The policy relies on maximizing the relative discrepancies among the Q-value estimates. Numerical experiments show superior performances of our exploration strategy

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keyzhu2019uncertainty

Related papers

• Smart Exploration In Reinforcement Learning Using Bounded Uncertainty Models (2025)0.00
• The Uncertainty Bellman Equation And Exploration (2017)0.00
• Model-based Epistemic Variance Of Values For Risk-aware Policy Optimization (2023)0.00
• How To Enable Uncertainty Estimation In Proximal Policy Optimization (2022)0.00
• A Nearly Optimal And Low-switching Algorithm For Reinforcement Learning With General Function Approximation (2023)0.00
• Statistical Bootstrapping For Uncertainty Estimation In Off-policy Evaluation (2020)0.00
• Estimating Risk And Uncertainty In Deep Reinforcement Learning (2019)0.00
• MEET: A Monte Carlo Exploration-exploitation Trade-off For Buffer Sampling (2022)2.26