Sample-efficient Reinforcement Learning With Stochastic Ensemble Value Expansion

Abstract

Integrating model-free and model-based approaches in reinforcement learning has the potential to achieve the high performance of model-free algorithms with low sample complexity. However, this is difficult because an imperfect dynamics model can degrade the performance of the learning algorithm, and in sufficiently complex environments, the dynamics model will almost always be imperfect. As a result, a key challenge is to combine model-based approaches with model-free learning in such a way that errors in the model do not degrade performance. We propose stochastic ensemble value expansion (STEVE), a novel model-based technique that addresses this issue. By dynamically interpolating between model rollouts of various horizon lengths for each individual example, STEVE ensures that the model is only utilized when doing so does not introduce significant errors. Our approach outperforms model-free baselines on challenging continuous control benchmarks with an order-of-magnitude increase in s

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybuckman2018sample

Related papers

• Model-based Value Estimation For Efficient Model-free Reinforcement Learning (2018)0.00
• Diminishing Return Of Value Expansion Methods In Model-based Reinforcement Learning (2023)0.00
• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00
• Sample Efficient Reinforcement Learning Via Model-ensemble Exploration And Exploitation (2021)0.00
• Efficient And Robust Reinforcement Learning With Uncertainty-based Value Expansion (2019)0.00
• Is Model Ensemble Necessary? Model-based RL Via A Single Model With Lipschitz Regularized Value Function (2023)0.00
• SEERL: Sample Efficient Ensemble Reinforcement Learning (2020)2.26
• Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach (2023)0.00