Bayesian Exploration Networks

Abstract

Bayesian reinforcement learning (RL) offers a principled and elegant approach for sequential decision making under uncertainty. Most notably, Bayesian agents do not face an exploration/exploitation dilemma, a major pathology of frequentist methods. However theoretical understanding of model-free approaches is lacking. In this paper, we introduce a novel Bayesian model-free formulation and the first analysis showing that model-free approaches can yield Bayes-optimal policies. We show all existing model-free approaches make approximations that yield policies that can be arbitrarily Bayes-suboptimal. As a first step towards model-free Bayes optimality, we introduce the Bayesian exploration network (BEN) which uses normalising flows to model both the aleatoric uncertainty (via density estimation) and epistemic uncertainty (via variational inference) in the Bellman operator. In the limit of complete optimisation, BEN learns true Bayes-optimal policies, but like in variational expectation-ma

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfellows2023bayesian

Related papers

• Smart Exploration In Reinforcement Learning Using Bounded Uncertainty Models (2025)0.00
• Online Bayesian Risk-averse Reinforcement Learning (2025)0.00
• Generalized Bayesian Deep Reinforcement Learning (2024)0.00
• Dynamic Subgoal-based Exploration Via Bayesian Optimization (2019)0.00
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• Efficient Exploration With Double Uncertain Value Networks (2017)0.00
• The Uncertainty Bellman Equation And Exploration (2017)0.00
• Bayesian Residual Policy Optimization: Scalable Bayesian Reinforcement Learning With Clairvoyant Experts (2020)0.00