Information-directed Exploration For Deep Reinforcement Learning

Abstract

Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynikolov2018information

Related papers

• STEERING: Stein Information Directed Exploration For Model-based Reinforcement Learning (2023)0.00
• Context-dependent Upper-confidence Bounds For Directed Exploration (2018)0.00
• Sampling Efficient Deep Reinforcement Learning Through Preference-guided Stochastic Exploration (2022)8.09
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• Effective Exploration For Deep Reinforcement Learning Via Bootstrapped Q-ensembles Under Tsallis Entropy Regularization (2018)0.00
• Exploration By Random Distribution Distillation (2025)0.00
• Langevin DQN (2020)0.00
• Decoupled Exploration And Exploitation Policies For Sample-efficient Reinforcement Learning (2021)0.00