Estimating Risk And Uncertainty In Deep Reinforcement Learning

Abstract

Reinforcement learning agents are faced with two types of uncertainty. Epistemic uncertainty stems from limited data and is useful for exploration, whereas aleatoric uncertainty arises from stochastic environments and must be accounted for in risk-sensitive applications. We highlight the challenges involved in simultaneously estimating both of them, and propose a framework for disentangling and estimating these uncertainties on learned Q-values. We derive unbiased estimators of these uncertainties and introduce an uncertainty-aware DQN algorithm, which we show exhibits safe learning behavior and outperforms other DQN variants on the MinAtar testbed.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyclements2019estimating

Related papers

• A Review Of Uncertainty For Deep Reinforcement Learning (2022)12.47
• Uncertainty-aware Low-rank Q-matrix Estimation For Deep Reinforcement Learning (2021)0.00
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• How To Enable Uncertainty Estimation In Proximal Policy Optimization (2022)0.00
• Uncertainty-based Out-of-distribution Detection In Deep Reinforcement Learning (2019)7.50
• Parameter-free Reduction Of The Estimation Bias In Deep Reinforcement Learning For Deterministic Policy Gradients (2021)0.00
• Query The Agent: Improving Sample Efficiency Through Epistemic Uncertainty Estimation (2022)0.00
• On Practical Robust Reinforcement Learning: Practical Uncertainty Set And Double-agent Algorithm (2023)3.58