Towards A Better Understanding Of Representation Dynamics Under Td-learning

Abstract

TD-learning is a foundation reinforcement learning (RL) algorithm for value prediction. Critical to the accuracy of value predictions is the quality of state representations. In this work, we consider the question: how does end-to-end TD-learning impact the representation over time? Complementary to prior work, we provide a set of analysis that sheds further light on the representation dynamics under TD-learning. We first show that when the environments are reversible, end-to-end TD-learning strictly decreases the value approximation error over time. Under further assumptions on the environments, we can connect the representation dynamics with spectral decomposition over the transition matrix. This latter finding establishes fitting multiple value functions from randomly generated rewards as a useful auxiliary task for representation learning, as we empirically validate on both tabular and Atari game suites.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2023towards

Related papers

• Learning Sparse Representations In Reinforcement Learning (2019)0.00
• TD Or Not TD: Analyzing The Role Of Temporal Differencing In Deep Reinforcement Learning (2018)0.00
• Discerning Temporal Difference Learning (2023)0.00
• Preferential Temporal Difference Learning (2021)0.00
• Representations For Stable Off-policy Reinforcement Learning (2020)0.00
• Prediction And Control In Continual Reinforcement Learning (2023)0.00
• On The Statistical Benefits Of Temporal Difference Learning (2023)0.00
• TD Convergence: An Optimization Perspective (2023)0.00