Discerning Temporal Difference Learning

Abstract

Temporal difference learning (TD) is a foundational concept in reinforcement learning (RL), aimed at efficiently assessing a policy's value function. TD(\(\lambda\)), a potent variant, incorporates a memory trace to distribute the prediction error into the historical context. However, this approach often neglects the significance of historical states and the relative importance of propagating the TD error, influenced by challenges such as visitation imbalance or outcome noise. To address this, we propose a novel TD algorithm named discerning TD learning (DTD), which allows flexible emphasis functions\(-\)predetermined or adapted during training\(-\)to allocate efforts effectively across states. We establish the convergence properties of our method within a specific class of emphasis functions and showcase its promising potential for adaptation to deep RL contexts. Empirical results underscore that employing a judicious emphasis function not only improves value estimation but also exped

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyma2023discerning

Related papers

• Preferential Temporal Difference Learning (2021)0.00
• TD Or Not TD: Analyzing The Role Of Temporal Differencing In Deep Reinforcement Learning (2018)0.00
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00
• Meta-learning Eligibility Traces For More Sample Efficient Temporal Difference Learning (2020)0.00
• Approximate Temporal Difference Learning Is A Gradient Descent For Reversible Policies (2018)0.00
• Control Theoretic Analysis Of Temporal Difference Learning (2021)0.00
• Prediction And Control In Continual Reinforcement Learning (2023)0.00
• Adaptive Temporal-difference Learning For Policy Evaluation With Per-state Uncertainty Estimates (2019)0.00