Model Selection In Batch Policy Optimization

Abstract

We study the problem of model selection in batch policy optimization: given a fixed, partial-feedback dataset and \(M\) model classes, learn a policy with performance that is competitive with the policy derived from the best model class. We formalize the problem in the contextual bandit setting with linear model classes by identifying three sources of error that any model selection algorithm should optimally trade-off in order to be competitive: (1) approximation error, (2) statistical complexity, and (3) coverage. The first two sources are common in model selection for supervised learning, where optimally trading-off these properties is well-studied. In contrast, the third source is unique to batch policy optimization and is due to dataset shift inherent to the setting. We first show that no batch policy optimization algorithm can achieve a guarantee addressing all three simultaneously, revealing a stark contrast between difficulties in batch policy optimization and the positive resul

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylee2021model

Related papers

• B2MAPO: A Batch-by-batch Multi-agent Policy Optimization To Balance Performance And Efficiency (2024)0.00
• When To Trust Your Model: Model-based Policy Optimization (2019)0.00
• Off-policy Evaluation And Learning From Logged Bandit Feedback: Error Reduction Via Surrogate Policy (2018)0.00
• Mismatched No More: Joint Model-policy Optimization For Model-based RL (2021)0.00
• How To Fine-tune The Model: Unified Model Shift And Model Bias Policy Optimization (2023)0.00
• Batch Size-invariance For Policy Optimization (2021)0.00
• Stackelberg Batch Policy Learning (2023)0.00
• Iterative Batch Reinforcement Learning Via Safe Diversified Model-based Policy Search (2024)0.00