Exploration Via Elliptical Episodic Bonuses

Abstract

In recent years, a number of reinforcement learning (RL) methods have been proposed to explore complex environments which differ across episodes. In this work, we show that the effectiveness of these methods critically relies on a count-based episodic term in their exploration bonus. As a result, despite their success in relatively simple, noise-free settings, these methods fall short in more realistic scenarios where the state space is vast and prone to noise. To address this limitation, we introduce Exploration via Elliptical Episodic Bonuses (E3B), a new method which extends count-based episodic bonuses to continuous state spaces and encourages an agent to explore states that are diverse under a learned embedding within each episode. The embedding is learned using an inverse dynamics model in order to capture controllable aspects of the environment. Our method sets a new state-of-the-art across 16 challenging tasks from the MiniHack suite, without requiring task-specific inductive b

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyhenaff2022exploration

Related papers

• DEIR: Efficient And Robust Exploration Through Discriminative-model-based Episodic Intrinsic Rewards (2023)0.00
• Learning-driven Exploration For Reinforcement Learning (2019)6.45
• Rewarding Episodic Visitation Discrepancy For Exploration In Reinforcement Learning (2022)0.00
• Episodic Reinforcement Learning With Expanded State-reward Space (2024)0.00
• Exploration In Feature Space For Reinforcement Learning (2017)0.00
• Fast Active Learning For Pure Exploration In Reinforcement Learning (2020)0.00
• Redeeming Intrinsic Rewards Via Constrained Optimization (2022)0.00
• Neighboring State-based Exploration For Reinforcement Learning (2022)0.00