Regularized Gradient Temporal-difference Learning

Abstract

Gradient temporal-difference (GTD) learning algorithms are widely used for off-policy policy evaluation with function approximation. However, existing convergence analyses rely on the restrictive assumption that the so-called feature interaction matrix (FIM) is nonsingular. In practice, the FIM can become singular and leads to instability or degraded performance. In this paper, we propose a regularized optimization objective by reformulating the mean-square projected Bellman error (MSPBE) minimization. This formulation naturally yields a regularized GTD algorithms, referred to as R-GTD, which guarantees convergence to a unique solution even when the FIM is singular. We establish theoretical convergence guarantees and explicit error bounds for the proposed method, and validate its effectiveness through empirical experiments.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyna2026regularized

Related papers

• New Versions Of Gradient Temporal Difference Learning (2021)0.00
• Finite-sample Analysis Of Proximal Gradient TD Algorithms (2020)0.00
• Proximal Gradient Temporal Difference Learning: Stable Reinforcement Learning With Polynomial Sample Complexity (2020)5.84
• Revisiting A Design Choice In Gradient Temporal Difference Learning (2023)0.00
• Gradient Iterated Temporal-difference Learning (2026)0.00
• Nonlinear Distributional Gradient Temporal-difference Learning (2018)0.00
• Backstepping Temporal Difference Learning (2023)0.00
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00