Control-oriented Model-based Reinforcement Learning With Implicit Differentiation

Abstract

The shortcomings of maximum likelihood estimation in the context of model-based reinforcement learning have been highlighted by an increasing number of papers. When the model class is misspecified or has a limited representational capacity, model parameters with high likelihood might not necessarily result in high performance of the agent on a downstream control task. To alleviate this problem, we propose an end-to-end approach for model learning which directly optimizes the expected returns using implicit differentiation. We treat a value function that satisfies the Bellman optimality operator induced by the model as an implicit function of model parameters and show how to differentiate the function. We provide theoretical and empirical evidence highlighting the benefits of our approach in the model misspecification regime compared to likelihood-based methods.

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keynikishin2021control

Related papers

• On The Model-based Stochastic Value Gradient For Continuous Reinforcement Learning (2020)0.00
• Value-biased Maximum Likelihood Estimation For Model-based Reinforcement Learning In Discounted Linear Mdps (2023)0.00
• On The Model-misspecification In Reinforcement Learning (2023)0.00
• Value Gradient Weighted Model-based Reinforcement Learning (2022)0.00
• The Value Equivalence Principle For Model-based Reinforcement Learning (2020)0.00
• Objective Mismatch In Model-based Reinforcement Learning (2020)0.00
• How To Fine-tune The Model: Unified Model Shift And Model Bias Policy Optimization (2023)0.00
• Between Rate-distortion Theory & Value Equivalence In Model-based Reinforcement Learning (2022)0.00