Why So Pessimistic? Estimating Uncertainties For Offline RL Through Ensembles, And Why Their Independence Matters
2022 · Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, Ofir Nachum
Abstract
Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of \(Q\)-functions can be leveraged as the primary source of pessimism for offline reinforcement learning (RL). We begin by identifying a critical flaw in a popular algorithmic choice used by many ensemble-based RL algorithms, namely the use of shared pessimistic target values when computing each ensemble member's Bellman error. Through theoretical analyses and construction of examples in toy MDPs, we demonstrate that shared pessimistic targets can paradoxically lead to value estimates that are effectively optimistic. Given this result, we propose MSG, a practical offline RL algorithm that trains an ensemble of \(Q\)-functions with independently computed targets based on completely separate networks, and optimizes a policy with respect to the lower confidence bound of predicted action values. Our experiments on the popular D4RL and RL Unplugged offline RL ben
Authors
(none)
Tags
Stats
Related papers
- Is Pessimism Provably Efficient For Offline RL? (2020)0.00
- Diverse Randomized Value Functions: A Provably Pessimistic Approach For Offline Reinforcement Learning (2024)3.58
- Uncertainty-based Offline Reinforcement Learning With Diversified Q-ensemble (2021)0.00
- Pessimistic Value Iteration For Multi-task Data Sharing In Offline Reinforcement Learning (2024)9.33
- Double Pessimism Is Provably Efficient For Distributionally Robust Offline Reinforcement Learning: Generic Algorithm And Robust Partial Coverage (2023)0.00
- Pessimistic Bootstrapping For Uncertainty-driven Offline Reinforcement Learning (2022)0.00
- Neural Network Approximation For Pessimistic Offline Reinforcement Learning (2023)0.00
- Model-based Offline Reinforcement Learning With Pessimism-modulated Dynamics Belief (2022)0.00