Asymptotically Unbiased Off-policy Policy Evaluation When Reusing Old Data In Nonstationary Environments

Abstract

In this work, we consider the off-policy policy evaluation problem for contextual bandits and finite horizon reinforcement learning in the nonstationary setting. Reusing old data is critical for policy evaluation, but existing estimators that reuse old data introduce large bias such that we can not obtain a valid confidence interval. Inspired from a related field called survey sampling, we introduce a variant of the doubly robust (DR) estimator, called the regression-assisted DR estimator, that can incorporate the past data without introducing a large bias. The estimator unifies several existing off-policy policy evaluation methods and improves on them with the use of auxiliary information and a regression approach. We prove that the new estimator is asymptotically unbiased, and provide a consistent variance estimator to a construct a large sample confidence interval. Finally, we empirically show that the new estimator improves estimation for the current and future policy values, and p

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2023asymptotically

Related papers

• Intrinsically Efficient, Stable, And Bounded Off-policy Evaluation For Reinforcement Learning (2019)0.00
• On The Reuse Bias In Off-policy Reinforcement Learning (2022)3.58
• Robust On-policy Sampling For Data-efficient Policy Evaluation In Reinforcement Learning (2021)0.00
• Doubly Optimal Policy Evaluation For Reinforcement Learning (2024)0.00
• Doubly Robust Off-policy Value And Gradient Estimation For Deterministic Policies (2020)0.00
• Online Estimation And Inference For Robust Policy Evaluation In Reinforcement Learning (2023)2.26
• Black-box Off-policy Estimation For Infinite-horizon Reinforcement Learning (2020)0.00
• Adaptive Temporal-difference Learning For Policy Evaluation With Per-state Uncertainty Estimates (2019)0.00