Gradient Iterated Temporal-difference Learning

Abstract

Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird's counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this work, we modify iterated TD l

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvincent2026gradient

Related papers

• Preferential Temporal Difference Learning (2021)0.00
• Discerning Temporal Difference Learning (2023)0.00
• O\(^2\)TD: (near)-optimal Off-policy TD Learning (2017)0.00
• Gradient Temporal-difference Learning With Regularized Corrections (2020)0.00
• Approximate Temporal Difference Learning Is A Gradient Descent For Reversible Policies (2018)0.00
• Revisiting A Design Choice In Gradient Temporal Difference Learning (2023)0.00
• Backstepping Temporal Difference Learning (2023)0.00
• Differential Temporal Difference Learning (2018)5.24