Off-policy Fitted Q-evaluation With Differentiable Function Approximators: Z-estimation And Inference Theory

Abstract

Off-Policy Evaluation (OPE) serves as one of the cornerstones in Reinforcement Learning (RL). Fitted Q Evaluation (FQE) with various function approximators, especially deep neural networks, has gained practical success. While statistical analysis has proved FQE to be minimax-optimal with tabular, linear and several nonparametric function families, its practical performance with more general function approximator is less theoretically understood. We focus on FQE with general differentiable function approximators, making our theory applicable to neural function approximations. We approach this problem using the Z-estimation theory and establish the following results: The FQE estimation error is asymptotically normal with explicit variance determined jointly by the tangent space of the function class at the ground truth, the reward structure, and the distribution shift due to off-policy learning; The finite-sample FQE error bound is dominated by the same variance term, and it can also be

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2022off

Related papers

• Bootstrapping Fitted Q-evaluation For Off-policy Inference (2021)0.00
• Fitted Q Evaluation Without Bellman Completeness Via Stationary Weighting (2025)0.00
• Minimax-optimal Off-policy Evaluation With Linear Function Approximation (2020)0.00
• Sample Complexity Of Nonparametric Off-policy Evaluation On Low-dimensional Manifolds Using Deep Networks (2022)0.00
• State-action Similarity-based Representations For Off-policy Evaluation (2023)1.20
• Variance-aware Off-policy Evaluation With Linear Function Approximation (2021)0.00
• More Efficient Off-policy Evaluation Through Regularized Targeted Learning (2019)0.00
• Interpretable Off-policy Evaluation In Reinforcement Learning By Highlighting Influential Transitions (2020)0.00