On Optimistic Versus Randomized Exploration In Reinforcement Learning

Abstract

We discuss the relative merits of optimistic and randomized approaches to exploration in reinforcement learning. Optimistic approaches presented in the literature apply an optimistic boost to the value estimate at each state-action pair and select actions that are greedy with respect to the resulting optimistic value function. Randomized approaches sample from among statistically plausible value functions and select actions that are greedy with respect to the random sample. Prior computational experience suggests that randomized approaches can lead to far more statistically efficient learning. We present two simple analytic examples that elucidate why this is the case. In principle, there should be optimistic approaches that fare well relative to randomized approaches, but that would require intractable computation. Optimistic approaches that have been proposed in the literature sacrifice statistical efficiency for the sake of computational efficiency. Randomized approaches, on the oth

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyosband2017on

Related papers

• Strategically Efficient Exploration In Competitive Multi-agent Reinforcement Learning (2021)0.00
• Exploration Conscious Reinforcement Learning Revisited (2018)0.00
• Probabilistic Insights For Efficient Exploration Strategies In Reinforcement Learning (2025)0.00
• Provably Efficient Exploration In Policy Optimization (2019)0.00
• Minimax-optimal Reward-agnostic Exploration In Reinforcement Learning (2023)0.00
• Conservative Exploration In Reinforcement Learning (2020)0.00
• Optimistic {\epsilon}-greedy Exploration For Cooperative Multi-agent Reinforcement Learning (2025)0.00
• Pitfall Of Optimism: Distributional Reinforcement Learning By Randomizing Risk Criterion (2023)0.00