Mismatched No More: Joint Model-policy Optimization For Model-based RL

Abstract

Many model-based reinforcement learning (RL) methods follow a similar template: fit a model to previously observed data, and then use data from that model for RL or planning. However, models that achieve better training performance (e.g., lower MSE) are not necessarily better for control: an RL agent may seek out the small fraction of states where an accurate model makes mistakes, or it might act in ways that do not expose the errors of an inaccurate model. As noted in prior work, there is an objective mismatch: models are useful if they yield good policies, but they are trained to maximize their accuracy, rather than the performance of the policies that result from them. In this work, we propose a single objective for jointly training the model and the policy, such that updates to either component increase a lower bound on expected return. To the best of our knowledge, this is the first lower bound for model-based RL that holds globally and can be efficiently estimated in continuous s

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyeysenbach2021mismatched

Related papers

• Simplifying Model-based RL: Learning Representations, Latent-space Models, And Policies With One Objective (2022)0.00
• How To Fine-tune The Model: Unified Model Shift And Model Bias Policy Optimization (2023)0.00
• When To Update Your Model: Constrained Model-based Reinforcement Learning (2022)2.26
• Objective Mismatch In Model-based Reinforcement Learning (2020)0.00
• The Virtues Of Laziness In Model-based RL: A Unified Objective And Algorithms (2023)0.00
• When To Trust Your Model: Model-based Policy Optimization (2019)0.00
• Robust Model-free Reinforcement Learning With Multi-objective Bayesian Optimization (2019)11.08
• On The Model-misspecification In Reinforcement Learning (2023)0.00