On The Model-misspecification In Reinforcement Learning

Abstract

The success of reinforcement learning (RL) crucially depends on effective function approximation when dealing with complex ground-truth models. Existing sample-efficient RL algorithms primarily employ three approaches to function approximation: policy-based, value-based, and model-based methods. However, in the face of model misspecification (a disparity between the ground-truth and optimal function approximators), it is shown that policy-based approaches can be robust even when the policy function approximation is under a large locally-bounded misspecification error, with which the function class may exhibit a \(Ω(1)\) approximation error in specific states and actions, but remains small on average within a policy-induced state distribution. Yet it remains an open question whether similar robustness can be achieved with value-based and model-based approaches, especially with general function approximation. To bridge this gap, in this paper we present a unified theoretical framework

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2023on

Related papers

• The Optimal Approximation Factors In Misspecified Off-policy Value Function Estimation (2023)0.00
• Reinforcement Learning Under Model Mismatch (2017)0.00
• Online Model Selection For Reinforcement Learning With Function Approximation (2020)0.00
• Misspecification In Inverse Reinforcement Learning (2022)5.24
• Quantifying The Sensitivity Of Inverse Reinforcement Learning To Misspecification (2024)0.00
• Learning The Reward Function For A Misspecified Model (2018)0.00
• Partial Identifiability And Misspecification In Inverse Reinforcement Learning (2024)0.00
• Online Robust Reinforcement Learning With Model Uncertainty (2021)0.00