Learning State Representations Via Retracing In Reinforcement Learning

Abstract

We propose learning via retracing, a novel self-supervised approach for learning the state representation (and the associated dynamics model) for reinforcement learning tasks. In addition to the predictive (reconstruction) supervision in the forward direction, we propose to include "retraced" transitions for representation / model learning, by enforcing the cycle-consistency constraint between the original and retraced states, hence improve upon the sample efficiency of learning. Moreover, learning via retracing explicitly propagates information about future transitions backward for inferring previous states, thus facilitates stronger representation learning for the downstream reinforcement learning tasks. We introduce Cycle-Consistency World Model (CCWM), a concrete model-based instantiation of learning via retracing. Additionally we propose a novel adaptive "truncation" mechanism for counteracting the negative impacts brought by "irreversible" transitions such that learning via retra

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyu2021learning

Related papers

• Recall Traces: Backtracking Models For Efficient Reinforcement Learning (2018)0.00
• Understanding Self-predictive Learning For Reinforcement Learning (2022)0.00
• Bootstrapped Representations In Reinforcement Learning (2023)0.00
• Value-consistent Representation Learning For Data-efficient Reinforcement Learning (2022)0.00
• Continual State Representation Learning For Reinforcement Learning Using Generative Replay (2018)0.00
• Continual Learning Using World Models For Pseudo-rehearsal (2019)0.00
• Playvirtual: Augmenting Cycle-consistent Virtual Trajectories For Reinforcement Learning (2021)0.00
• There Is No Turning Back: A Self-supervised Approach For Reversibility-aware Reinforcement Learning (2021)0.00