High-confidence Error Estimates For Learned Value Functions

Abstract

Estimating the value function for a fixed policy is a fundamental problem in reinforcement learning. Policy evaluation algorithms---to estimate value functions---continue to be developed, to improve convergence rates, improve stability and handle variability, particularly for off-policy learning. To understand the properties of these algorithms, the experimenter needs high-confidence estimates of the accuracy of the learned value functions. For environments with small, finite state-spaces, like chains, the true value function can be easily computed, to compute accuracy. For large, or continuous state-spaces, however, this is no longer feasible. In this paper, we address the largely open problem of how to obtain these high-confidence estimates, for general state-spaces. We provide a high-confidence bound on an empirical estimate of the value error to the true value error. We use this bound to design an offline sampling algorithm, which stores the required quantities to repeatedly comput

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysajed2018high

Related papers

• Chaining Value Functions For Off-policy Learning (2022)0.00
• Statistical Inference Of The Value Function For Reinforcement Learning In Infinite Horizon Settings (2020)13.14
• Statistical Bootstrapping For Uncertainty Estimation In Off-policy Evaluation (2020)0.00
• Hybrid Value Estimation For Off-policy Evaluation And Offline Reinforcement Learning (2022)0.00
• Confidence-conditioned Value Functions For Offline Reinforcement Learning (2022)0.00
• Kalman Meets Bellman: Improving Policy Evaluation Through Value Tracking (2020)0.00
• On The Limited Representational Power Of Value Functions And Its Links To Statistical (in)efficiency (2024)0.00
• Foresee Then Evaluate: Decomposing Value Estimation With Latent Future Prediction (2021)3.58