Objective Mismatch In Model-based Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) has been shown to be a powerful framework for data-efficiently learning control of continuous tasks. Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, with little development of the general framework. In this paper, we identify a fundamental issue of the standard MBRL framework -- what we call the objective mismatch issue. Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, metric will also be optimized. In the context of MBRL, we characterize the objective mismatch between training the forward dynamics model w.r.t.~the likelihood of the one-step ahead prediction, and the overall goal of improving performance on a downstream control task. For example, this issue can emerge with the realization that dynamics models effective for a specific task do not necessarily need to be globally accurate, and vice versa globally accurate models might

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylambert2020objective

Related papers

• Mismatched No More: Joint Model-policy Optimization For Model-based RL (2021)0.00
• Plan To Predict: Learning An Uncertainty-foreseeing Model For Model-based Reinforcement Learning (2023)0.00
• Planning With Exploration: Addressing Dynamics Bottleneck In Model-based Reinforcement Learning (2020)0.00
• The Virtues Of Laziness In Model-based RL: A Unified Objective And Algorithms (2023)0.00
• How To Fine-tune The Model: Unified Model Shift And Model Bias Policy Optimization (2023)0.00
• Robust Model-free Reinforcement Learning With Multi-objective Bayesian Optimization (2019)11.08
• When To Update Your Model: Constrained Model-based Reinforcement Learning (2022)2.26
• Model Imitation For Model-based Reinforcement Learning (2019)0.00