Offline Bayesian Aleatoric And Epistemic Uncertainty Quantification And Posterior Value Optimisation In Finite-state Mdps

Abstract

We address the challenge of quantifying Bayesian uncertainty and incorporating it in offline use cases of finite-state Markov Decision Processes (MDPs) with unknown dynamics. Our approach provides a principled method to disentangle epistemic and aleatoric uncertainty, and a novel technique to find policies that optimise Bayesian posterior expected value without relying on strong assumptions about the MDP's posterior distribution. First, we utilise standard Bayesian reinforcement learning methods to capture the posterior uncertainty in MDP parameters based on available data. We then analytically compute the first two moments of the return distribution across posterior samples and apply the law of total variance to disentangle aleatoric and epistemic uncertainties. To find policies that maximise posterior expected value, we leverage the closed-form expression for value as a function of policy. This allows us to propose a stochastic gradient-based approach for solving the problem. We illu

Stats

• citations0
• S2 citations—
• github stars2
• HF likes0
• heat score0.95
• arxiv keyvaldettaro2024offline

Related papers

• Bayesian Learning Of The Optimal Action-value Function In A Markov Decision Process (2025)0.00
• Bayesian Risk-sensitive Policy Optimization For Mdps With General Loss Functions (2025)0.00
• An Offline Risk-aware Policy Selection Method For Bayesian Markov Decision Processes (2021)0.00
• Bayesian Learning Of Optimal Policies In Markov Decision Processes With Countably Infinite State-space (2023)0.00
• Model-based Epistemic Variance Of Values For Risk-aware Policy Optimization (2023)0.00
• Bayesian Policy Optimization For Model Uncertainty (2018)0.00
• Deterministic Uncertainty Propagation For Improved Model-based Offline Reinforcement Learning (2024)0.00
• Model-free Learning And Optimal Policy Design In Multi-agent Mdps Under Probabilistic Agent Dropout (2023)2.26