Enhancing Offline Model-based RL Via Active Model Selection: A Bayesian Optimization Perspective

Abstract

Offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data with the help of learned dynamics models. To fully unleash the power of offline MBRL, model selection plays a pivotal role in determining the dynamics model utilized for downstream policy learning. However, offline MBRL conventionally relies on validation or off-policy evaluation, which are rather inaccurate due to the inherent distribution shift in offline RL. To tackle this, we propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget, through the lens of Bayesian optimization (BO). Specifically, we recast model selection as BO and enable probabilistic inference in BOMS by proposing a novel model-induced kernel, which is theoretically grounded and computationally efficient. Through extensive experiments, we show that BOMS improves over the baseline me

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyang2025enhancing

Related papers

• Bayes Adaptive Monte Carlo Tree Search For Offline Model-based Reinforcement Learning (2024)0.00
• Conservative Bayesian Model-based Value Expansion For Offline Policy Optimization (2022)0.00
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00
• Morel : Model-based Offline Reinforcement Learning (2020)0.00
• Policy-driven World Model Adaptation For Robust Offline Model-based Reinforcement Learning (2025)0.00
• Overcoming Model Bias For Robust Offline Deep Reinforcement Learning (2020)11.58
• Offline-boosted Actor-critic: Adaptively Blending Optimal Historical Behaviors In Deep Off-policy RL (2024)0.00
• BECAUSE: Bilinear Causal Representation For Generalizable Offline Model-based Reinforcement Learning (2024)0.00