Auxiliary Task Discovery Through Generate-and-test

Abstract

In this paper, we explore an approach to auxiliary task discovery in reinforcement learning based on ideas from representation learning. Auxiliary tasks tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task of maximizing reward, and thus producing better representations. Typically these tasks are designed by people. Meta-learning offers a promising avenue for automatic task discovery; however, these methods are computationally expensive and challenging to tune in practice. In this paper, we explore a complementary approach to the auxiliary task discovery: continually generating new auxiliary tasks and preserving only those with high utility. We also introduce a new measure of auxiliary tasks' usefulness based on how useful the features induced by them are for the main task. Our discovery algorithm significantly outperforms random tasks and learning without auxiliary tasks across a suite of environments.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyrafiee2022auxiliary

Related papers

• Proto-value Networks: Scaling Representation Learning With Auxiliary Tasks (2023)0.00
• Continual Auxiliary Task Learning (2022)0.00
• What Makes Useful Auxiliary Tasks In Reinforcement Learning: Investigating The Effect Of The Target Policy (2022)0.00
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• On The Effect Of Auxiliary Tasks On Representation Dynamics (2021)0.00
• When Does Self-prediction Help? Understanding Auxiliary Tasks In Reinforcement Learning (2024)0.00
• Return-based Contrastive Representation Learning For Reinforcement Learning (2021)12.17
• Ensemble And Auxiliary Tasks For Data-efficient Deep Reinforcement Learning (2021)0.00