Sample Complexity And Overparameterization Bounds For Temporal Difference Learning With Neural Network Approximation

Abstract

In this paper, we study the dynamics of temporal difference learning with neural network-based value function approximation over a general state space, namely, *Neural TD learning*. We consider two practically used algorithms, projection-free and max-norm regularized Neural TD learning, and establish the first convergence bounds for these algorithms. An interesting observation from our results is that max-norm regularization can dramatically improve the performance of TD learning algorithms, both in terms of sample complexity and overparameterization. In particular, we prove that max-norm regularization improves state-of-the-art sample complexity and overparameterization bounds. The results in this work rely on a novel Lyapunov drift analysis of the network parameters as a stopped and controlled random process.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycayci2021sample

Related papers

• An Improved Finite-time Analysis Of Temporal Difference Learning With Deep Neural Networks (2024)0.00
• On The Performance Of Temporal Difference Learning With Neural Networks (2023)0.00
• Geometric Insights Into The Convergence Of Nonlinear TD Learning (2019)0.00
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00
• Neural Temporal-difference And Q-learning Provably Converge To Global Optima (2019)7.81
• A Finite Time Analysis Of Temporal Difference Learning With Linear Function Approximation (2018)0.00
• Temporal-difference Learning With Nonlinear Function Approximation: Lazy Training And Mean Field Regimes (2019)0.00
• Differential Temporal Difference Learning (2018)5.24