Statistical Bootstrapping For Uncertainty Estimation In Off-policy Evaluation

Abstract

In reinforcement learning, it is typical to use the empirically observed transitions and rewards to estimate the value of a policy via either model-based or Q-fitting approaches. Although straightforward, these techniques in general yield biased estimates of the true value of the policy. In this work, we investigate the potential for statistical bootstrapping to be used as a way to take these biased estimates and produce calibrated confidence intervals for the true value of the policy. We identify conditions - specifically, sufficient data size and sufficient coverage - under which statistical bootstrapping in this setting is guaranteed to yield correct confidence intervals. In practical situations, these conditions often do not hold, and so we discuss and propose mechanisms that can be employed to mitigate their effects. We evaluate our proposed method and show that it can yield accurate confidence intervals in a variety of conditions, including challenging continuous control environm

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykostrikov2020statistical

Related papers

• Bootstrapping With Models: Confidence Intervals For Off-policy Evaluation (2016)9.23
• Bootstrapping Fitted Q-evaluation For Off-policy Inference (2021)0.00
• Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning (2021)9.23
• Pessimistic Bootstrapping For Uncertainty-driven Offline Reinforcement Learning (2022)0.00
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• High-confidence Error Estimates For Learned Value Functions (2018)0.00
• Towards Robust Off-policy Learning For Runtime Uncertainty (2022)0.00
• Online Estimation And Inference For Robust Policy Evaluation In Reinforcement Learning (2023)2.26