OER: Offline Experience Replay For Continual Offline Reinforcement Learning

Abstract

The capability of continuously learning new skills via a sequence of pre-collected offline datasets is desired for an agent. However, consecutively learning a sequence of offline tasks likely leads to the catastrophic forgetting issue under resource-limited scenarios. In this paper, we formulate a new setting, continual offline reinforcement learning (CORL), where an agent learns a sequence of offline reinforcement learning tasks and pursues good performance on all learned tasks with a small replay buffer without exploring any of the environments of all the sequential tasks. For consistently learning on all sequential tasks, an agent requires acquiring new knowledge and meanwhile preserving old knowledge in an offline manner. To this end, we introduced continual learning algorithms and experimentally found experience replay (ER) to be the most suitable algorithm for the CORL problem. However, we observe that introducing ER into CORL encounters a new distribution shift problem: the mism

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keygai2023oer

Related papers

• Solving Continual Offline Reinforcement Learning With Decision Transformer (2024)0.00
• CORL: Research-oriented Deep Offline Reinforcement Learning Library (2022)0.00
• Replay-enhanced Continual Reinforcement Learning (2023)0.00
• Tsn-affinity: Similarity-driven Parameter Reuse For Continual Offline Reinforcement Learning (2026)0.00
• Adaptive Replay Buffer For Offline-to-online Reinforcement Learning (2025)0.00
• Continual Offline Reinforcement Learning Via Diffusion-based Dual Generative Replay (2024)0.00
• Experience Replay For Continual Learning (2018)0.00
• OPAL: Offline Primitive Discovery For Accelerating Offline Reinforcement Learning (2020)0.00