Replay-enhanced Continual Reinforcement Learning

Abstract

Replaying past experiences has proven to be a highly effective approach for averting catastrophic forgetting in supervised continual learning. However, some crucial factors are still largely ignored, making it vulnerable to serious failure, when used as a solution to forgetting in continual reinforcement learning, even in the context of perfect memory where all data of previous tasks are accessible in the current task. On the one hand, since most reinforcement learning algorithms are not invariant to the reward scale, the previously well-learned tasks (with high rewards) may appear to be more salient to the current learning process than the current task (with small initial rewards). This causes the agent to concentrate on those salient tasks at the expense of generality on the current task. On the other hand, offline learning on replayed tasks while learning a new task may induce a distributional shift between the dataset and the learned policy on old tasks, resulting in forgetting. In

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2023replay

Related papers

• Experience Replay For Continual Learning (2018)0.00
• Augmented Replay Memory In Reinforcement Learning With Continuous Control (2019)5.24
• Stable Continual Reinforcement Learning Via Diffusion-based Trajectory Replay (2024)0.00
• Learn The Time To Learn: Replay Scheduling In Continual Learning (2022)0.00
• Map-based Experience Replay: A Memory-efficient Solution To Catastrophic Forgetting In Reinforcement Learning (2023)9.23
• Replay For Safety (2021)0.00
• Remember And Forget For Experience Replay (2018)0.00
• Continual Offline Reinforcement Learning Via Diffusion-based Dual Generative Replay (2024)0.00