Geometric Insights Into The Convergence Of Nonlinear TD Learning

Abstract

While there are convergence guarantees for temporal difference (TD) learning when using linear function approximators, the situation for nonlinear models is far less understood, and divergent examples are known. Here we take a first step towards extending theoretical convergence guarantees to TD learning with nonlinear function approximation. More precisely, we consider the expected learning dynamics of the TD(0) algorithm for value estimation. As the step-size converges to zero, these dynamics are defined by a nonlinear ODE which depends on the geometry of the space of function approximators, the structure of the underlying Markov chain, and their interaction. We find a set of function approximators that includes ReLU networks and has geometry amenable to TD learning regardless of environment, so that the solution performs about as well as linear TD in the worst case. Then, we show how environments that are more reversible induce dynamics that are better for TD learning and prove glob

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybrandfonbrener2019geometric

Related papers

• TD Convergence: An Optimization Perspective (2023)0.00
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00
• Single-timescale Stochastic Nonconvex-concave Optimization For Smooth Nonlinear TD Learning (2020)0.00
• Analysis Of Off-policy \(n\)-step Td-learning With Linear Function Approximation (2025)0.00
• Neural Temporal-difference And Q-learning Provably Converge To Global Optima (2019)7.81
• Finite Sample Analysis Of Linear Temporal Difference Learning With Arbitrary Features (2025)0.00
• Convergence Of TD(0) Under Polynomial Mixing With Nonlinear Function Approximation (2025)0.00
• A Finite Time Analysis Of Temporal Difference Learning With Linear Function Approximation (2018)0.00