Statistical Inference Of The Value Function For Reinforcement Learning In Infinite Horizon Settings

Abstract

Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision making problems. The goodness of a policy is measured by its value function starting from some initial state. The focus of this paper is to construct confidence intervals (CIs) for a policy's value in infinite horizon settings where the number of decision points diverges to infinity. We propose to model the action-value state function (Q-function) associated with a policy based on series/sieve method to derive its confidence interval. When the target policy depends on the observed data as well, we propose a SequentiAl Value Evaluation (SAVE) method to recursively update the estimated policy and its value estimator. As long as either the number of trajectories or the number of decision points diverges to infinity, we show that the proposed CI achieves nominal coverage even in cases where the optimal policy is not unique. Simulation studies

Stats

• citations34
• S2 citations—
• github stars5
• HF likes0
• heat score13.14
• arxiv keyshi2020statistical

Related papers

• High-confidence Error Estimates For Learned Value Functions (2018)0.00
• Value-distributional Model-based Reinforcement Learning (2023)1.56
• On The Continuity And Smoothness Of The Value Function In Reinforcement Learning And Optimal Control (2024)0.00
• On The Convergence Of Policy Iteration-based Reinforcement Learning With Monte Carlo Policy Evaluation (2023)0.00
• A Policy Gradient Approach For Finite Horizon Constrained Markov Decision Processes (2022)3.58
• Uncertainty Quantification And Exploration For Reinforcement Learning (2019)6.77
• The Value-improvement Path: Towards Better Representations For Reinforcement Learning (2020)6.77
• Confounding-robust Policy Evaluation In Infinite-horizon Reinforcement Learning (2020)0.00