The Surprising Ineffectiveness Of Pre-trained Visual Representations For Model-based Reinforcement Learning
2024 · Moritz Schneider, Robert Krug, Narunas Vaskevicius, et al.
Abstract
Visual Reinforcement Learning (RL) methods often require extensive amounts of data. As opposed to model-free RL, model-based RL (MBRL) offers a potential solution with efficient data utilization through planning. Additionally, RL lacks generalization capabilities for real-world tasks. Prior work has shown that incorporating pre-trained visual representations (PVRs) enhances sample efficiency and generalization. While PVRs have been extensively studied in the context of model-free RL, their potential in MBRL remains largely unexplored. In this paper, we benchmark a set of PVRs on challenging control tasks in a model-based RL setting. We investigate the data efficiency, generalization capabilities, and the impact of different properties of PVRs on the performance of model-based agents. Our results, perhaps surprisingly, reveal that for MBRL current PVRs are not more sample efficient than learning representations from scratch, and that they do not generalize better to out-of-distribution
Authors
(none)
Tags
Stats
Related papers
- Value-consistent Representation Learning For Data-efficient Reinforcement Learning (2022)0.00
- Investigating Pre-training Objectives For Generalization In Vision-based Reinforcement Learning (2024)2.16
- The Role Of Pretrained Representations For The OOD Generalization Of Reinforcement Learning Agents (2021)0.00
- Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
- On The Reliability And Generalizability Of Brain-inspired Reinforcement Learning Algorithms (2020)0.00
- Machine Versus Human Attention In Deep Reinforcement Learning Tasks (2020)0.00
- Improving Sample Efficiency In Model-free Reinforcement Learning From Images (2019)16.99
- Good Actions Succeed, Bad Actions Generalize: A Case Study On Why RL Generalizes Better (2025)0.00