When Does Self-prediction Help? Understanding Auxiliary Tasks In Reinforcement Learning

Abstract

We investigate the impact of auxiliary learning tasks such as observation reconstruction and latent self-prediction on the representation learning problem in reinforcement learning. We also study how they interact with distractions and observation functions in the MDP. We provide a theoretical analysis of the learning dynamics of observation reconstruction, latent self-prediction, and TD learning in the presence of distractions and observation functions under linear model assumptions. With this formalization, we are able to explain why latent-self prediction is a helpful *auxiliary task*, while observation reconstruction can provide more useful features when used in isolation. Our empirical analysis shows that the insights obtained from our learning dynamics framework predicts the behavior of these loss functions beyond the linear model assumption in non-linear neural networks. This reinforces the usefulness of the linear model framework not only for theoretical analysis, but also prac

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvoelcker2024when

Related papers

• Continual Auxiliary Task Learning (2022)0.00
• On The Effect Of Auxiliary Tasks On Representation Dynamics (2021)0.00
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• What Makes Useful Auxiliary Tasks In Reinforcement Learning: Investigating The Effect Of The Target Policy (2022)0.00
• Work In Progress: Temporally Extended Auxiliary Tasks (2020)0.00
• Understanding Self-predictive Learning For Reinforcement Learning (2022)0.00
• Auxiliary Task Discovery Through Generate-and-test (2022)0.00
• Proto-value Networks: Scaling Representation Learning With Auxiliary Tasks (2023)0.00