Representation Matters: Offline Pretraining For Sequential Decision Making

Abstract

The recent success of supervised learning methods on ever larger offline datasets has spurred interest in the reinforcement learning (RL) field to investigate whether the same paradigms can be translated to RL algorithms. This research area, known as offline RL, has largely focused on offline policy optimization, aiming to find a return-maximizing policy exclusively from offline data. In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We aim to answer the question, what unsupervised objectives applied to offline datasets are able to learn state representations which elevate performance on downstream tasks, whether those downstream tasks be online RL, imitation learning from expert demonstrations, or even offline policy optimization based on the same offline dataset? Through a variety of experiments utilizing standard offline RL datasets, we find that the use of pretraining with unsupervised learning objectives can dra

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyang2021representation

Related papers

• Behavior Prior Representation Learning For Offline Reinforcement Learning (2022)0.00
• Using Offline Data To Speed Up Reinforcement Learning In Procedurally Generated Environments (2023)6.77
• Towards Data-driven Offline Simulations For Online Reinforcement Learning (2022)0.00
• Rvs: What Is Essential For Offline RL Via Supervised Learning? (2021)0.00
• An Optimistic Perspective On Offline Reinforcement Learning (2019)0.00
• Expert-supervised Reinforcement Learning For Offline Policy Learning And Evaluation (2020)0.00
• Bridging The Gap Between Offline And Online Reinforcement Learning Evaluation Methodologies (2022)0.00
• A Policy-guided Imitation Approach For Offline Reinforcement Learning (2022)0.00