Map-based Experience Replay: A Memory-efficient Solution To Catastrophic Forgetting In Reinforcement Learning

Abstract

Deep Reinforcement Learning agents often suffer from catastrophic forgetting, forgetting previously found solutions in parts of the input space when training on new data. Replay Memories are a common solution to the problem, decorrelating and shuffling old and new training samples. They naively store state transitions as they come in, without regard for redundancy. We introduce a novel cognitive-inspired replay memory approach based on the Grow-When-Required (GWR) self-organizing network, which resembles a map-based mental model of the world. Our approach organizes stored transitions into a concise environment-model-like network of state-nodes and transition-edges, merging similar samples to reduce the memory size and increase pair-wise distance among samples, which increases the relevancy of each sample. Overall, our paper shows that map-based experience replay allows for significant memory reduction with only small performance decreases.

Stats

• citations16
• S2 citations—
• github stars0
• HF likes0
• heat score9.23
• arxiv keyhafez2023map

Related papers

• Augmented Replay Memory In Reinforcement Learning With Continuous Control (2019)5.24
• Replay-enhanced Continual Reinforcement Learning (2023)0.00
• A Deeper Look At Experience Replay (2017)0.00
• Experience Replay For Continual Learning (2018)0.00
• Prioritized Generative Replay (2024)0.00
• Replay Buffer With Local Forgetting For Adapting To Local Environment Changes In Deep Model-based Reinforcement Learning (2023)0.00
• Stabilising Experience Replay For Deep Multi-agent Reinforcement Learning (2017)0.00
• Remember And Forget For Experience Replay (2018)0.00