Revisiting A Design Choice In Gradient Temporal Difference Learning

Abstract

Off-policy learning enables a reinforcement learning (RL) agent to reason counterfactually about policies that are not executed and is one of the most important ideas in RL. It, however, can lead to instability when combined with function approximation and bootstrapping, two arguably indispensable ingredients for large-scale reinforcement learning. This is the notorious deadly triad. The seminal work Sutton et al. (2008) pioneers Gradient Temporal Difference learning (GTD) as the first solution to the deadly triad, which has enjoyed massive success thereafter. During the derivation of GTD, some intermediate algorithm, called \(A^\top\)TD, was invented but soon deemed inferior. In this paper, we revisit this \(A^\top\)TD and prove that a variant of \(A^\top\)TD, called \(A_t^\top\)TD, is also an effective solution to the deadly triad. Furthermore, this \(A_t^\top\)TD only needs one set of parameters and one learning rate. By contrast, GTD has two sets of parameters and two learning rate

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyqian2023revisiting

Related papers

• New Versions Of Gradient Temporal Difference Learning (2021)0.00
• Backstepping Temporal Difference Learning (2023)0.00
• Discerning Temporal Difference Learning (2023)0.00
• Approximate Temporal Difference Learning Is A Gradient Descent For Reversible Policies (2018)0.00
• Gradient Iterated Temporal-difference Learning (2026)0.00
• Proximal Gradient Temporal Difference Learning: Stable Reinforcement Learning With Polynomial Sample Complexity (2020)5.84
• Regularized Gradient Temporal-difference Learning (2026)0.00
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00