A First Empirical Study Of Emphatic Temporal Difference Learning

Abstract

In this paper we present the first empirical study of the emphatic temporal-difference learning algorithm (ETD), comparing it with conventional temporal-difference learning, in particular, with linear TD(0), on on-policy and off-policy variations of the Mountain Car problem. The initial motivation for developing ETD was that it has good convergence properties under off-policy training (Sutton, Mahmood and White 2016), but it is also a new algorithm for the on-policy case. In both our on-policy and off-policy experiments, we found that each method converged to a characteristic asymptotic level of error, with ETD better than TD(0). TD(0) achieved a still lower error level temporarily before falling back to its higher asymptote, whereas ETD never showed this kind of "bounce". In the off-policy case (in which TD(0) is not guaranteed to converge), ETD was significantly slower.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyghiassian2017a

Related papers

• Should All Temporal Difference Learning Use Emphasis? (2019)0.00
• PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method (2021)0.00
• Discerning Temporal Difference Learning (2023)0.00
• Truncated Emphatic Temporal Difference Methods For Prediction And Control (2021)0.00
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00
• Preferential Temporal Difference Learning (2021)0.00
• Backstepping Temporal Difference Learning (2023)0.00
• O\(^2\)TD: (near)-optimal Off-policy TD Learning (2017)0.00