When Should We Prefer State-to-visual Dagger Over Visual Reinforcement Learning?

Abstract

Learning policies from high-dimensional visual inputs, such as pixels and point clouds, is crucial in various applications. Visual reinforcement learning is a promising approach that directly trains policies from visual observations, although it faces challenges in sample efficiency and computational costs. This study conducts an empirical comparison of State-to-Visual DAgger, a two-stage framework that initially trains a state policy before adopting online imitation to learn a visual policy, and Visual RL across a diverse set of tasks. We evaluate both methods across 16 tasks from three benchmarks, focusing on their asymptotic performance, sample efficiency, and computational costs. Surprisingly, our findings reveal that State-to-Visual DAgger does not universally outperform Visual RL but shows significant advantages in challenging tasks, offering more consistent performance. In contrast, its benefits in sample efficiency are less pronounced, although it often reduces the overall wall

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymu2024when

Related papers

• Value-consistent Representation Learning For Data-efficient Reinforcement Learning (2022)0.00
• Challenges And Opportunities In Offline Reinforcement Learning From Visual Observations (2022)0.00
• Understanding What Affects The Generalization Gap In Visual Reinforcement Learning: Theory And Empirical Evidence (2024)5.84
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• A Study On Dense And Sparse (visual) Rewards In Robot Policy Learning (2021)0.00
• A Recipe For Unbounded Data Augmentation In Visual Reinforcement Learning (2024)0.00
• Learning To Identify Critical States For Reinforcement Learning From Videos (2023)8.76
• Visual Processing In Context Of Reinforcement Learning (2022)0.00