Ensemble Successor Representations For Task Generalization In Offline-to-online Reinforcement Learning

Abstract

In Reinforcement Learning (RL), training a policy from scratch with online experiences can be inefficient because of the difficulties in exploration. Recently, offline RL provides a promising solution by giving an initialized offline policy, which can be refined through online interactions. However, existing approaches primarily perform offline and online learning in the same task, without considering the task generalization problem in offline-to-online adaptation. In real-world applications, it is common that we only have an offline dataset from a specific task while aiming for fast online-adaptation for several tasks. To address this problem, our work builds upon the investigation of successor representations for task generalization in online RL and extends the framework to incorporate offline-to-online learning. We demonstrate that the conventional paradigm using successor features cannot effectively utilize offline data and improve the performance for the new task by online fine-tu

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keywang2024ensemble

Related papers

• The Generalization Gap In Offline Reinforcement Learning (2023)0.00
• Representation Matters: Offline Pretraining For Sequential Decision Making (2021)0.00
• Improving Zero-shot Generalization In Offline Reinforcement Learning Using Generalized Similarity Functions (2021)2.26
• Boosting Offline Reinforcement Learning With Residual Generative Modeling (2021)0.00
• Successor Feature Sets: Generalizing Successor Representations Across Policies (2021)5.84
• A Non-monolithic Policy Approach Of Offline-to-online Reinforcement Learning (2024)0.00
• A Policy-guided Imitation Approach For Offline Reinforcement Learning (2022)0.00
• Using Offline Data To Speed Up Reinforcement Learning In Procedurally Generated Environments (2023)6.77