Oracle Inequalities For Model Selection In Offline Reinforcement Learning

Abstract

In offline reinforcement learning (RL), a learner leverages prior logged data to learn a good policy without interacting with the environment. A major challenge in applying such methods in practice is the lack of both theoretically principled and practical tools for model selection and evaluation. To address this, we study the problem of model selection in offline RL with value function approximation. The learner is given a nested sequence of model classes to minimize squared Bellman error and must select among these to achieve a balance between approximation and estimation error of the classes. We propose the first model selection algorithm for offline RL that achieves minimax rate-optimal oracle inequalities up to logarithmic factors. The algorithm, ModBE, takes as input a collection of candidate model classes and a generic base offline RL algorithm. By successively eliminating model classes using a novel one-sided generalization test, ModBE returns a policy with regret scaling with

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylee2022oracle

Related papers

• Model-based Reinforcement Learning With Double Oracle Efficiency In Policy Optimization And Offline Estimation (2026)0.00
• Pessimistic Nonlinear Least-squares Value Iteration For Offline Reinforcement Learning (2023)0.00
• Conservative Bayesian Model-based Value Expansion For Offline Policy Optimization (2022)0.00
• Morel : Model-based Offline Reinforcement Learning (2020)0.00
• Enhancing Offline Model-based RL Via Active Model Selection: A Bayesian Optimization Perspective (2025)0.00
• Is Value Learning Really The Main Bottleneck In Offline RL? (2024)0.00
• Deployment-efficient Reinforcement Learning Via Model-based Offline Optimization (2020)0.00
• Optimality Inductive Biases And Agnostic Guidelines For Offline Reinforcement Learning (2021)0.00