Hyperparameter Optimization Can Even Be Harmful In Off-policy Learning And How To Deal With It

Abstract

There has been a growing interest in off-policy evaluation in the literature such as recommender systems and personalized medicine. We have so far seen significant progress in developing estimators aimed at accurately estimating the effectiveness of counterfactual policies based on biased logged data. However, there are many cases where those estimators are used not only to evaluate the value of decision making policies but also to search for the best hyperparameters from a large candidate space. This work explores the latter hyperparameter optimization (HPO) task for off-policy learning. We empirically show that naively applying an unbiased estimator of the generalization performance as a surrogate objective in HPO can cause an unexpected failure, merely pursuing hyperparameters whose generalization performance is greatly overestimated. We then propose simple and computationally efficient corrections to the typical HPO procedure to deal with the aforementioned issues simultaneously. E

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysaito2024hyperparameter

Related papers

• Hyperparameter Optimisation With Practical Interpretability And Explanation Methods In Probabilistic Curriculum Learning (2025)0.00
• Online Hyper-parameter Tuning In Off-policy Learning Via Evolutionary Strategies (2020)0.00
• A Method For Evaluating Hyperparameter Sensitivity In Reinforcement Learning (2024)0.00
• Hyperparameter Selection For Imitation Learning (2021)0.00
• Policy Optimization As Online Learning With Mediator Feedback (2020)0.00
• Fast Efficient Hyperparameter Tuning For Policy Gradients (2019)0.00
• Revisiting Design Choices In Proximal Policy Optimization (2020)0.00
• Automating Control Of Overestimation Bias For Reinforcement Learning (2021)0.00