CLARE: Conservative Model-based Reward Learning For Offline Inverse Reinforcement Learning

Abstract

This work aims to tackle a major challenge in offline Inverse Reinforcement Learning (IRL), namely the reward extrapolation error, where the learned reward function may fail to explain the task correctly and misguide the agent in unseen environments due to the intrinsic covariate shift. Leveraging both expert data and lower-quality diverse data, we devise a principled algorithm (namely CLARE) that solves offline IRL efficiently via integrating "conservatism" into a learned reward function and utilizing an estimated dynamics model. Our theoretical analysis provides an upper bound on the return gap between the learned policy and the expert policy, based on which we characterize the impact of covariate shift by examining subtle two-tier tradeoffs between the exploitation (on both expert and diverse data) and exploration (on the estimated dynamics model). We show that CLARE can provably alleviate the reward extrapolation error by striking the right exploitation-exploration balance therein.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyue2023clare

Related papers

• Offline Inverse RL: New Solution Concepts And Provably Efficient Algorithms (2024)0.00
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework For Offline Inverse Reinforcement Learning (2023)0.00
• DOMAIN: Mildly Conservative Model-based Offline Reinforcement Learning (2023)0.00
• Long-horizon Model-based Offline Reinforcement Learning Without Conservatism (2025)0.00
• A Bayesian Approach To Robust Inverse Reinforcement Learning (2023)0.00
• Conservative Bayesian Model-based Value Expansion For Offline Policy Optimization (2022)0.00
• Bridging Offline Reinforcement Learning And Imitation Learning: A Tale Of Pessimism (2021)0.00
• Learning From Sparse Offline Datasets Via Conservative Density Estimation (2024)0.00