The Benefits Of Model-based Generalization In Reinforcement Learning

Abstract

Model-Based Reinforcement Learning (RL) is widely believed to have the potential to improve sample efficiency by allowing an agent to synthesize large amounts of imagined experience. Experience Replay (ER) can be considered a simple kind of model, which has proved effective at improving the stability and efficiency of deep RL. In principle, a learned parametric model could improve on ER by generalizing from real experience to augment the dataset with additional plausible experience. However, given that learned value functions can also generalize, it is not immediately obvious why model generalization should be better. Here, we provide theoretical and empirical insight into when, and how, we can expect data generated by a learned model to be useful. First, we provide a simple theorem motivating how learning a model as an intermediate step can narrow down the set of possible value functions more than learning a value function directly from data using the Bellman equation. Second, we prov

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyoung2022the

Related papers

• When To Use Parametric Models In Reinforcement Learning? (2019)0.00
• On The Power Of Pre-training For Generalization In RL: Provable Benefits And Hardness (2022)0.00
• When To Trust Your Model: Model-based Policy Optimization (2019)0.00
• Improving Generalization In Meta Reinforcement Learning Using Learned Objectives (2019)0.00
• The Value Equivalence Principle For Model-based Reinforcement Learning (2020)0.00
• The Principle Of Unchanged Optimality In Reinforcement Learning Generalization (2019)0.00
• Measuring And Characterizing Generalization In Deep Reinforcement Learning (2018)9.76
• Assessing Generalization In Deep Reinforcement Learning (2018)0.00