CCLF: A Contrastive-curiosity-driven Learning Framework For Sample-efficient Reinforcement Learning

Abstract

In reinforcement learning (RL), it is challenging to learn directly from high-dimensional observations, where data augmentation has recently been shown to remedy this via encoding invariances from raw pixels. Nevertheless, we empirically find that not all samples are equally important and hence simply injecting more augmented inputs may instead cause instability in Q-learning. In this paper, we approach this problem systematically by developing a model-agnostic Contrastive-Curiosity-Driven Learning Framework (CCLF), which can fully exploit sample importance and improve learning efficiency in a self-supervised manner. Facilitated by the proposed contrastive curiosity, CCLF is capable of prioritizing the experience replay, selecting the most informative augmented inputs, and more importantly regularizing the Q-function as well as the encoder to concentrate more on under-learned data. Moreover, it encourages the agent to explore with a curiosity-based reward. As a result, the agent can fo

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keysun2022cclf

Related papers

• Using Contrastive Samples For Identifying And Leveraging Possible Causal Relationships In Reinforcement Learning (2022)0.00
• Contrastive UCB: Provably Efficient Contrastive Self-supervised Learning In Online Reinforcement Learning (2022)0.00
• Human-inspired Framework To Accelerate Reinforcement Learning (2023)0.00
• Curriculum Learning For Reinforcement Learning Domains: A Framework And Survey (2020)0.00
• Improving Context-based Meta-reinforcement Learning With Self-supervised Trajectory Contrastive Learning (2021)0.00
• Contrastive Learning As Goal-conditioned Reinforcement Learning (2022)0.00
• A Survey On Enhancing Reinforcement Learning In Complex Environments: Insights From Human And LLM Feedback (2024)0.00
• CUDC: A Curiosity-driven Unsupervised Data Collection Method With Adaptive Temporal Distances For Offline Reinforcement Learning (2023)2.26