Continual Reinforcement Learning By Planning With Online World Models

Abstract

Continual reinforcement learning (CRL) refers to a naturalistic setting where an agent needs to endlessly evolve, by trial and error, to solve multiple tasks that are presented sequentially. One of the largest obstacles to CRL is that the agent may forget how to solve previous tasks when learning a new task, known as catastrophic forgetting. In this paper, we propose to address this challenge by planning with online world models. Specifically, we learn a Follow-The-Leader shallow model online to capture the world dynamics, in which we plan using model predictive control to solve a set of tasks specified by any reward functions. The online world model is immune to forgetting by construction with a proven regret bound of \(\mathcal\{O\}(\sqrt\{K^2Dlog(T)\})\) under mild assumptions. The planner searches actions solely based on the latest online model, thus forming a FTL Online Agent (OA) that updates incrementally. To assess OA, we further design Continual Bench, a dedicated environment

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2025continual

Related papers

• Continual Policy Distillation From Distributed Reinforcement Learning Teachers (2026)0.00
• Continual Visual Reinforcement Learning With A Life-long World Model (2023)2.26
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021)0.00
• Demonstration-guided Continual Reinforcement Learning In Dynamic Environments (2025)0.00
• Task-agnostic Continual Reinforcement Learning: Gaining Insights And Overcoming Challenges (2022)0.00
• Online Reinforcement Learning In Non-stationary Context-driven Environments (2023)0.00
• Simple Recipe Works: Vision-language-action Models Are Natural Continual Learners With Reinforcement Learning (2026)0.00
• Continual Knowledge Adaptation For Reinforcement Learning (2025)0.00