Proto-value Networks: Scaling Representation Learning With Auxiliary Tasks

Abstract

Auxiliary tasks improve the representations learned by deep reinforcement learning agents. Analytically, their effect is reasonably well understood; in practice, however, their primary use remains in support of a main learning objective, rather than as a method for learning representations. This is perhaps surprising given that many auxiliary tasks are defined procedurally, and hence can be treated as an essentially infinite source of information about the environment. Based on this observation, we study the effectiveness of auxiliary tasks for learning rich representations, focusing on the setting where the number of tasks and the size of the agent's network are simultaneously increased. For this purpose, we derive a new family of auxiliary tasks based on the successor measure. These tasks are easy to implement and have appealing theoretical properties. Combined with a suitable off-policy learning rule, the result is a representation learning algorithm that can be understood as extend

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfarebrother2023proto

Related papers

• What Makes Useful Auxiliary Tasks In Reinforcement Learning: Investigating The Effect Of The Target Policy (2022)0.00
• Auxiliary Task Discovery Through Generate-and-test (2022)0.00
• On The Effect Of Auxiliary Tasks On Representation Dynamics (2021)0.00
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• Continual Auxiliary Task Learning (2022)0.00
• Return-based Contrastive Representation Learning For Reinforcement Learning (2021)12.17
• When Does Self-prediction Help? Understanding Auxiliary Tasks In Reinforcement Learning (2024)0.00
• Work In Progress: Temporally Extended Auxiliary Tasks (2020)0.00