Introspective Experience Replay: Look Back When Surprised

Abstract

In reinforcement learning (RL), experience replay-based sampling techniques play a crucial role in promoting convergence by eliminating spurious correlations. However, widely used methods such as uniform experience replay (UER) and prioritized experience replay (PER) have been shown to have sub-optimal convergence and high seed sensitivity respectively. To address these issues, we propose a novel approach called IntrospectiveExperience Replay (IER) that selectively samples batches of data points prior to surprising events. Our method builds upon the theoretically sound reverse experience replay (RER) technique, which has been shown to reduce bias in the output of Q-learning-type algorithms with linear function approximation. However, this approach is not always practical or reliable when using neural function approximation. Through empirical evaluations, we demonstrate that IER with neural function approximation yields reliable and superior performance compared toUER, PER, and hindsigh

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykumar2022introspective

Related papers

• Replay For Safety (2021)0.00
• Large Batch Experience Replay (2021)0.00
• Remember And Forget For Experience Replay (2018)0.00
• Reward Prediction Error Prioritisation In Experience Replay: The RPE-PER Method (2025)0.00
• Prioritized Sequence Experience Replay (2019)0.00
• Regret Minimization Experience Replay In Off-policy Reinforcement Learning (2021)0.00
• Learning To Sample With Local And Global Contexts In Experience Replay Buffer (2020)0.00
• A Deeper Look At Experience Replay (2017)0.00