Online Bootstrap Inference For Policy Evaluation In Reinforcement Learning

Abstract

The recent emergence of reinforcement learning has created a demand for robust statistical inference methods for the parameter estimates computed using these algorithms. Existing methods for statistical inference in online learning are restricted to settings involving independently sampled observations, while existing statistical inference methods in reinforcement learning (RL) are limited to the batch setting. The online bootstrap is a flexible and efficient approach for statistical inference in linear stochastic approximation algorithms, but its efficacy in settings involving Markov noise, such as RL, has yet to be explored. In this paper, we study the use of the online bootstrap method for statistical inference in RL. In particular, we focus on the temporal difference (TD) learning and Gradient TD (GTD) learning algorithms, which are themselves special instances of linear stochastic approximation under Markov noise. The method is shown to be distributionally consistent for statistic

Stats

• citations16
• S2 citations—
• github stars0
• HF likes0
• heat score9.23
• arxiv keyramprasad2021online

Related papers

• Online Estimation And Inference For Robust Policy Evaluation In Reinforcement Learning (2023)2.26
• Statistical Bootstrapping For Uncertainty Estimation In Off-policy Evaluation (2020)0.00
• Stochastic Gradient Descent With Dependent Data For Offline Reinforcement Learning (2022)0.00
• Finite Sample Analysis Of The GTD Policy Evaluation Algorithms In Markov Setting (2018)0.00
• Doubly Robust Interval Estimation For Optimal Policy Evaluation In Online Learning (2021)0.00
• Pessimistic Bootstrapping For Uncertainty-driven Offline Reinforcement Learning (2022)0.00
• Bootstrap Advantage Estimation For Policy Optimization In Reinforcement Learning (2022)0.00
• Bootstrapping Fitted Q-evaluation For Off-policy Inference (2021)0.00