Improving Sample Efficiency In Model-free Reinforcement Learning From Images

Abstract

Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a latent representation together with the control policy. However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance. Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning. However, incorporating reconstruction loss into an off-policy learning algorithm often leads to training instability. We explore the underlying reasons and identify variational autoencoders, used by previous investigations, as the cause of the divergence. Following these findings, we propose effective techniques to improve training stability. This results in a simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. Furthermore, our approach demonstrates robust

Stats

• citations183
• S2 citations—
• github stars0
• HF likes0
• heat score16.99
• arxiv keyyarats2019improving

Related papers

• Simplifying Model-based RL: Learning Representations, Latent-space Models, And Policies With One Objective (2022)0.00
• Model-free Reinforcement Learning For Model-based Control: Towards Safe, Interpretable And Sample-efficient Agents (2025)0.00
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• Iqrl -- Implicitly Quantized Representations For Sample-efficient Reinforcement Learning (2024)0.00
• Optimal Control With Natural Images: Efficient Reinforcement Learning Using Overcomplete Sparse Codes (2024)0.00
• Representation Learning For Efficient Deep Multi-agent Reinforcement Learning (2024)0.00
• Learning Off-policy With Model-based Intrinsic Motivation For Active Online Exploration (2024)0.00
• Accelerating Representation Learning With View-consistent Dynamics In Data-efficient Reinforcement Learning (2022)0.00