Work In Progress: Temporally Extended Auxiliary Tasks

Abstract

Predictive auxiliary tasks have been shown to improve performance in numerous reinforcement learning works, however, this effect is still not well understood. The primary purpose of the work presented here is to investigate the impact that an auxiliary task's prediction timescale has on the agent's policy performance. We consider auxiliary tasks which learn to make on-policy predictions using temporal difference learning. We test the impact of prediction timescale using a specific form of auxiliary task in which the input image is used as the prediction target, which we refer to as temporal difference autoencoders (TD-AE). We empirically evaluate the effect of TD-AE on the A2C algorithm in the VizDoom environment using different prediction timescales. While we do not observe a clear relationship between the prediction timescale on performance, we make the following observations: 1) using auxiliary tasks allows us to reduce the trajectory length of the A2C algorithm, 2) in some cases te

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysherstan2020work

Related papers

• Continual Auxiliary Task Learning (2022)0.00
• On The Effect Of Auxiliary Tasks On Representation Dynamics (2021)0.00
• What Makes Useful Auxiliary Tasks In Reinforcement Learning: Investigating The Effect Of The Target Policy (2022)0.00
• When Does Self-prediction Help? Understanding Auxiliary Tasks In Reinforcement Learning (2024)0.00
• Proto-value Networks: Scaling Representation Learning With Auxiliary Tasks (2023)0.00
• Auxiliary Task Discovery Through Generate-and-test (2022)0.00
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• Ensemble And Auxiliary Tasks For Data-efficient Deep Reinforcement Learning (2021)0.00