Policy-driven World Model Adaptation For Robust Offline Model-based Reinforcement Learning

Abstract

Offline reinforcement learning (RL) offers a powerful paradigm for data-driven control. Compared to model-free approaches, offline model-based RL (MBRL) explicitly learns a world model from a static dataset and uses it as a surrogate simulator, improving data efficiency and enabling potential generalization beyond the dataset support. However, most existing offline MBRL methods follow a two-stage training procedure: first learning a world model by maximizing the likelihood of the observed transitions, then optimizing a policy to maximize its expected return under the learned model. This objective mismatch results in a world model that is not necessarily optimized for effective policy learning. Moreover, we observe that policies learned via offline MBRL often lack robustness during deployment, and small adversarial noise in the environment can lead to significant performance degradation. To address these, we propose a framework that dynamically adapts the world model alongside the polic

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2025policy

Related papers

• Bayes Adaptive Monte Carlo Tree Search For Offline Model-based Reinforcement Learning (2024)0.00
• Morel : Model-based Offline Reinforcement Learning (2020)0.00
• Enhancing Offline Model-based RL Via Active Model Selection: A Bayesian Optimization Perspective (2025)0.00
• Conservative Bayesian Model-based Value Expansion For Offline Policy Optimization (2022)0.00
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00
• A Workflow For Offline Model-free Robotic Reinforcement Learning (2021)0.00
• Model-based Offline Reinforcement Learning With Adversarial Data Augmentation (2025)0.00
• Overcoming Model Bias For Robust Offline Deep Reinforcement Learning (2020)11.58