Morel : Model-based Offline Reinforcement Learning

Abstract

In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. The ability to train RL policies offline can greatly expand the applicability of RL, its data efficiency, and its experimental velocity. Prior work in offline RL has been confined almost exclusively to model-free RL approaches. In this work, we present MOReL, an algorithmic framework for model-based offline RL. This framework consists of two steps: (a) learning a pessimistic MDP (P-MDP) using the offline dataset; and (b) learning a near-optimal policy in this P-MDP. The learned P-MDP has the property that for any policy, the performance in the real environment is approximately lower-bounded by the performance in the P-MDP. This enables it to serve as a good surrogate for purposes of policy evaluation and learning, and overcome common pitfalls of model-based RL like model exploitation. Theoretically, we show that MOReL is minim

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykidambi2020morel

Related papers

• Overcoming Model Bias For Robust Offline Deep Reinforcement Learning (2020)11.58
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00
• Model-based Offline Reinforcement Learning With Pessimism-modulated Dynamics Belief (2022)0.00
• Policy-driven World Model Adaptation For Robust Offline Model-based Reinforcement Learning (2025)0.00
• POPO: Pessimistic Offline Policy Optimization (2020)5.24
• Model-based Offline Reinforcement Learning With Adversarial Data Augmentation (2025)0.00
• Bridging Offline Reinforcement Learning And Imitation Learning: A Tale Of Pessimism (2021)0.00
• Conservative Bayesian Model-based Value Expansion For Offline Policy Optimization (2022)0.00