Rewarding Episodic Visitation Discrepancy For Exploration In Reinforcement Learning

Abstract

Exploration is critical for deep reinforcement learning in complex environments with high-dimensional observations and sparse rewards. To address this problem, recent approaches proposed to leverage intrinsic rewards to improve exploration, such as novelty-based exploration and prediction-based exploration. However, many intrinsic reward modules require sophisticated structures and representation learning, resulting in prohibitive computational complexity and unstable performance. In this paper, we propose Rewarding Episodic Visitation Discrepancy (REVD), a computation-efficient and quantified exploration method. More specifically, REVD provides intrinsic rewards by evaluating the R\'enyi divergence-based visitation discrepancy between episodes. To make efficient divergence estimation, a k-nearest neighbor estimator is utilized with a randomly-initialized state encoder. Finally, the REVD is tested on Atari games and PyBullet Robotics Environments. Extensive experiments demonstrate that

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyuan2022rewarding

Related papers

• DEIR: Efficient And Robust Exploration Through Discriminative-model-based Episodic Intrinsic Rewards (2023)0.00
• Long-term Visitation Value For Deep Exploration In Sparse Reward Reinforcement Learning (2020)7.24
• Never Explore Repeatedly In Multi-agent Reinforcement Learning (2023)0.00
• Exploration Via Elliptical Episodic Bonuses (2022)3.58
• R\'enyi State Entropy For Exploration Acceleration In Reinforcement Learning (2022)0.00
• Self-supervised Exploration Via Temporal Inconsistency In Reinforcement Learning (2022)3.58
• Information Content Exploration (2023)0.00
• Redeeming Intrinsic Rewards Via Constrained Optimization (2022)0.00