Meta-learning Eligibility Traces For More Sample Efficient Temporal Difference Learning

Abstract

Temporal-Difference (TD) learning is a standard and very successful reinforcement learning approach, at the core of both algorithms that learn the value of a given policy, as well as algorithms which learn how to improve policies. TD-learning with eligibility traces provides a way to do temporal credit assignment, i.e. decide which portion of a reward should be assigned to predecessor states that occurred at different previous times, controlled by a parameter \(\lambda\). However, tuning this parameter can be time-consuming, and not tuning it can lead to inefficient learning. To improve the sample efficiency of TD-learning, we propose a meta-learning method for adjusting the eligibility trace parameter, in a state-dependent manner. The adaptation is achieved with the help of auxiliary learners that learn distributional information about the update targets online, incurring roughly the same computational complexity per step as the usual value learner. Our approach can be used both in on

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhao2020meta

Related papers

• Meta-learning State-based Eligibility Traces For More Sample-efficient Policy Evaluation (2019)0.00
• Discerning Temporal Difference Learning (2023)0.00
• Preferential Temporal Difference Learning (2021)0.00
• On Generalized Bellman Equations And Temporal-difference Learning (2017)5.84
• A Finite Time Analysis Of Temporal Difference Learning With Linear Function Approximation (2018)0.00
• On The Statistical Benefits Of Temporal Difference Learning (2023)0.00
• Adaptive Temporal-difference Learning For Policy Evaluation With Per-state Uncertainty Estimates (2019)0.00
• Adaptive Temporal Difference Learning With Linear Function Approximation (2020)0.00