The Role Of Pretrained Representations For The OOD Generalization Of Reinforcement Learning Agents
2021 · Andrea Dittadi, Frederik Träuble, Manuel Wüthrich, et al.
Abstract
Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pr
Authors
(none)
Tags
Stats
Related papers
- Rethinking Out-of-distribution Detection For Reinforcement Learning: Advancing Methods For Evaluation And Detection (2024)2.26
- An Information-theoretic Analysis Of OOD Generalization In Meta-reinforcement Learning (2025)0.00
- The Surprising Ineffectiveness Of Pre-trained Visual Representations For Model-based Reinforcement Learning (2024)0.00
- On The Power Of Pre-training For Generalization In RL: Provable Benefits And Hardness (2022)0.00
- Good Actions Succeed, Bad Actions Generalize: A Case Study On Why RL Generalizes Better (2025)0.00
- When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning (2022)0.00
- Investigating Pre-training Objectives For Generalization In Vision-based Reinforcement Learning (2024)2.16
- Assessing Generalization In Deep Reinforcement Learning (2018)0.00