Model-based Value Estimation For Efficient Model-free Reinforcement Learning

Abstract

Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of continuous control tasks. Unfortunately, they rely on heuristics that limit usage of the dynamics model. We present model-based value expansion, which controls for uncertainty in the model by only allowing imagination to fixed depth. By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfeinberg2018model

Related papers

• Diminishing Return Of Value Expansion Methods In Model-based Reinforcement Learning (2023)0.00
• Efficient And Robust Reinforcement Learning With Uncertainty-based Value Expansion (2019)0.00
• Sample-efficient Reinforcement Learning With Stochastic Ensemble Value Expansion (2018)0.00
• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00
• Is Model Ensemble Necessary? Model-based RL Via A Single Model With Lipschitz Regularized Value Function (2023)0.00
• Deciding What To Model: Value-equivalent Sampling For Reinforcement Learning (2022)0.00
• Efficient Exploration In Continuous-time Model-based Reinforcement Learning (2023)0.00
• Value-biased Maximum Likelihood Estimation For Model-based Reinforcement Learning In Discounted Linear Mdps (2023)0.00