A Deeper Look At Experience Replay

Abstract

Recently experience replay is widely used in various deep reinforcement learning (RL) algorithms, in this paper we rethink the utility of experience replay. It introduces a new hyper-parameter, the memory buffer size, which needs carefully tuning. However unfortunately the importance of this new hyper-parameter has been underestimated in the community for a long time. In this paper we did a systematic empirical study of experience replay under various function representations. We showcase that a large replay buffer can significantly hurt the performance. Moreover, we propose a simple O(1) method to remedy the negative influence of a large replay buffer. We showcase its utility in both simple grid world and challenging domains like Atari games.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2017a

Related papers

• Replay For Safety (2021)0.00
• Large Batch Experience Replay (2021)0.00
• Map-based Experience Replay: A Memory-efficient Solution To Catastrophic Forgetting In Reinforcement Learning (2023)9.23
• Improving Experience Replay With Successor Representation (2021)0.00
• CUER: Corrected Uniform Experience Replay For Off-policy Continuous Deep Reinforcement Learning Algorithms (2024)0.00
• Introspective Experience Replay: Look Back When Surprised (2022)0.00
• Bootstrapping A DQN Replay Memory With Synthetic Experiences (2020)5.84
• Stabilising Experience Replay For Deep Multi-agent Reinforcement Learning (2017)0.00