Continual Auxiliary Task Learning

Abstract

Learning auxiliary tasks, such as multiple predictions about the world, can provide many benefits to reinforcement learning systems. A variety of off-policy learning algorithms have been developed to learn such predictions, but as yet there is little work on how to adapt the behavior to gather useful data for those off-policy predictions. In this work, we investigate a reinforcement learning system designed to learn a collection of auxiliary tasks, with a behavior policy learning to take actions to improve those auxiliary predictions. We highlight the inherent non-stationarity in this continual auxiliary task learning problem, for both prediction learners and the behavior learner. We develop an algorithm based on successor features that facilitates tracking under non-stationary rewards, and prove the separation into learning successor features and rewards provides convergence rate improvements. We conduct an in-depth study into the resulting multi-prediction learning system.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymcleod2022continual

Related papers

• What Makes Useful Auxiliary Tasks In Reinforcement Learning: Investigating The Effect Of The Target Policy (2022)0.00
• When Does Self-prediction Help? Understanding Auxiliary Tasks In Reinforcement Learning (2024)0.00
• Work In Progress: Temporally Extended Auxiliary Tasks (2020)0.00
• Auxiliary Task Discovery Through Generate-and-test (2022)0.00
• Proto-value Networks: Scaling Representation Learning With Auxiliary Tasks (2023)0.00
• Return-based Contrastive Representation Learning For Reinforcement Learning (2021)12.17
• Continual Deep Reinforcement Learning With Task-agnostic Policy Distillation (2024)0.00
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26