Analysis Of Off-policy \(n\)-step Td-learning With Linear Function Approximation

Abstract

This paper analyzes multi-step temporal difference (TD)-learning algorithms within the ``deadly triad'' scenario, characterized by linear function approximation, off-policy learning, and bootstrapping. In particular, we prove that \(n\)-step TD-learning algorithms converge to a solution as the sampling horizon \(n\) increases sufficiently. The paper is divided into two parts. In the first part, we comprehensively examine the fundamental properties of their model-based deterministic counterparts, including projected value iteration, gradient descent algorithms, which can be viewed as prototype deterministic algorithms whose analysis plays a pivotal role in understanding and developing their model-free reinforcement learning counterparts. In particular, we prove that these algorithms converge to meaningful solutions when \(n\) is sufficiently large. Based on these findings, in the second part, two \(n\)-step TD-learning algorithms are proposed and analyzed, which can be seen as the model

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylim2025analysis

Related papers

• Analysis Of Off-policy Multi-step Td-learning With Linear Function Approximation (2024)2.26
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00
• A Finite Time Analysis Of Temporal Difference Learning With Linear Function Approximation (2018)0.00
• Backstepping Temporal Difference Learning (2023)0.00
• Geometric Insights Into The Convergence Of Nonlinear TD Learning (2019)0.00
• Single-timescale Stochastic Nonconvex-concave Optimization For Smooth Nonlinear TD Learning (2020)0.00
• Two Time-scale Off-policy TD Learning: Non-asymptotic Analysis Over Markovian Samples (2019)0.00
• Finite-time Performance Of Distributed Temporal Difference Learning With Linear Function Approximation (2019)9.59