Extending Differential Temporal Difference Methods For Episodic Problems

Abstract

arXiv:2605.04368v1 Announce Type: new Abstract: Differential temporal difference (TD) methods are value-based reinforcement learning algorithms that have been proposed for infinite-horizon problems. They rely on reward centering, where each reward is centered by the average reward. This keeps the return bounded and removes a value function's state-independent offset. However, reward centering can alter the optimal policy in episodic problems, limiting its applicability. Motivated by recent works that emphasize the role of normalization in streaming deep reinforcement learning, we study reward centering in episodic problems and propose a generalization of differential TD. We prove that this generalization maintains the ordering of policies in the presence of termination, and thus extends differential TD to episodic problems. We show equivalence with a form of linear TD, thereby inheriting theoretical guarantees that have been shown for those algorithms. We then extend several streaming

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyasis2026extending

Related papers

• Discerning Temporal Difference Learning (2023)0.00
• Differential Temporal Difference Learning (2018)5.24
• On The Statistical Benefits Of Temporal Difference Learning (2023)0.00
• Preferential Temporal Difference Learning (2021)0.00
• Control Theoretic Analysis Of Temporal Difference Learning (2021)0.00
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00
• Adaptive Temporal-difference Learning For Policy Evaluation With Per-state Uncertainty Estimates (2019)0.00
• Finite-time Performance Of Distributed Temporal Difference Learning With Linear Function Approximation (2019)9.59