Optimal Mixture Weights For Off-policy Evaluation With Multiple Behavior Policies

Abstract

Off-policy evaluation is a key component of reinforcement learning which evaluates a target policy with offline data collected from behavior policies. It is a crucial step towards safe reinforcement learning and has been used in advertisement, recommender systems and many other applications. In these applications, sometimes the offline data is collected from multiple behavior policies. Previous works regard data from different behavior policies equally. Nevertheless, some behavior policies are better at producing good estimators while others are not. This paper starts with discussing how to correctly mix estimators produced by different behavior policies. We propose three ways to reduce the variance of the mixture estimator when all sub-estimators are unbiased or asymptotically unbiased. Furthermore, experiments on simulated recommender systems show that our methods are effective in reducing the Mean-Square Error of estimation.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylai2020optimal

Related papers

• Behaviour Policy Optimization: Provably Lower Variance Return Estimates For Off-policy Reinforcement Learning (2025)0.00
• Infinite-horizon Off-policy Policy Evaluation With Multiple Behavior Policies (2019)0.00
• Combining Parametric And Nonparametric Models For Off-policy Evaluation (2019)0.00
• Projected State-action Balancing Weights For Offline Reinforcement Learning (2021)0.00
• Importance Sampling Policy Evaluation With An Estimated Behavior Policy (2018)0.00
• Doubly Optimal Policy Evaluation For Reinforcement Learning (2024)0.00
• Data-efficient Policy Evaluation Through Behavior Policy Search (2017)0.00
• Low Variance Off-policy Evaluation With State-based Importance Sampling (2022)0.00