Adaptive Temporal Difference Learning With Linear Function Approximation

Abstract

This paper revisits the temporal difference (TD) learning algorithm for the policy evaluation tasks in reinforcement learning. Typically, the performance of TD(0) and TD(\(\lambda\)) is very sensitive to the choice of stepsizes. Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link between the TD(0) learning algorithm and the stochastic gradient methods, we develop a provably convergent adaptive projected variant of the TD(0) learning algorithm with linear function approximation that we term AdaTD(0). In contrast to the TD(0), AdaTD(0) is robust or less sensitive to the choice of stepsizes. Analytically, we establish that to reach an \(\epsilon\) accuracy, the number of iterations needed is \(\tilde\{O\}(\epsilon^\{-2\}\ln^4\frac\{1\}\{\epsilon\}/\ln^4\frac\{1\}\{\rho\})\) in the general case, where \(\rho\) represents the speed of the underlying Markov chain converges to the stationary distribution. This implies that the iteration complexity of AdaTD(0) is no wo

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysun2020adaptive

Related papers

• A Finite Time Analysis Of Temporal Difference Learning With Linear Function Approximation (2018)0.00
• Finite-time Performance Of Distributed Temporal Difference Learning With Linear Function Approximation (2019)9.59
• Backstepping Temporal Difference Learning (2023)0.00
• Approximate Temporal Difference Learning Is A Gradient Descent For Reversible Policies (2018)0.00
• Analysis Of Off-policy \(n\)-step Td-learning With Linear Function Approximation (2025)0.00
• Discerning Temporal Difference Learning (2023)0.00
• Finite Sample Analysis Of Linear Temporal Difference Learning With Arbitrary Features (2025)0.00
• Finite-sample Analysis Of Decentralized Temporal-difference Learning With Linear Function Approximation (2019)0.00