Model-based Epistemic Variance Of Values For Risk-aware Policy Optimization

Abstract

We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over Markov decision processes (MDPs). Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation (UBE), but the over-approximation may result in inefficient exploration. We propose a new UBE whose solution converges to the true posterior variance over values and leads to lower regret in tabular exploration problems. We identify challenges to apply the UBE theory beyond tabular problems and propose a suitable approximation. Based on this approximation, we introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC), that can be applied for either risk-seeking or risk-averse policy optimization with minimal changes. Experiments in both online and offline RL demonstrate improved perf

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyluis2023model

Related papers

• Value-distributional Model-based Reinforcement Learning (2023)1.56
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• The Uncertainty Bellman Equation And Exploration (2017)0.00
• Deep Model-based Reinforcement Learning Via Estimated Uncertainty And Conservative Policy Optimization (2019)0.00
• Offline Bayesian Aleatoric And Epistemic Uncertainty Quantification And Posterior Value Optimisation In Finite-state Mdps (2024)0.95
• Bayesian Policy Optimization For Model Uncertainty (2018)0.00
• How To Enable Uncertainty Estimation In Proximal Policy Optimization (2022)0.00
• Smart Exploration In Reinforcement Learning Using Bounded Uncertainty Models (2025)0.00