Return-based Contrastive Representation Learning For Reinforcement Learning

Abstract

Recently, various auxiliary tasks have been proposed to accelerate representation learning and improve sample efficiency in deep reinforcement learning (RL). However, existing auxiliary tasks do not take the characteristics of RL problems into consideration and are unsupervised. By leveraging returns, the most important feedback signals in RL, we propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns. Our auxiliary loss is theoretically justified to learn representations that capture the structure of a new form of state-action abstraction, under which state-action pairs with similar return distributions are aggregated together. In low data regime, our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite, and achieves even better performance when combined with existing auxiliary tasks.

Stats

• citations41
• S2 citations—
• github stars0
• HF likes0
• heat score12.17
• arxiv keyliu2021return

Related papers

• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• Contrastive Learning As Goal-conditioned Reinforcement Learning (2022)0.00
• Intrinsically Motivated Self-supervised Learning In Reinforcement Learning (2021)3.58
• TACO: Temporal Latent Action-driven Contrastive Loss For Visual Reinforcement Learning (2023)0.00
• Proto-value Networks: Scaling Representation Learning With Auxiliary Tasks (2023)0.00
• Learning State Representations From Random Deep Action-conditional Predictions (2021)0.00
• Ensemble And Auxiliary Tasks For Data-efficient Deep Reinforcement Learning (2021)0.00
• Auxiliary Task Discovery Through Generate-and-test (2022)0.00