Understanding Self-predictive Learning For Reinforcement Learning

Abstract

We study the learning dynamics of self-predictive learning for reinforcement learning, a family of algorithms that learn representations by minimizing the prediction error of their own future latent representations. Despite its recent empirical success, such algorithms have an apparent defect: trivial representations (such as constants) minimize the prediction error, yet it is obviously undesirable to converge to such solutions. Our central insight is that careful designs of the optimization dynamics are critical to learning meaningful representations. We identify that a faster paced optimization of the predictor and semi-gradient updates on the representation, are crucial to preventing the representation collapse. Then in an idealized setup, we show self-predictive learning dynamics carries out spectral decomposition on the state transition matrix, effectively capturing information of the transition dynamics. Building on the theoretical insights, we propose bidirectional self-predicti

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytang2022understanding

Related papers

• Data-efficient Reinforcement Learning With Self-predictive Representations (2020)0.00
• Bridging State And History Representations: Understanding Self-predictive RL (2024)0.00
• A Unifying Framework For Action-conditional Self-predictive Reinforcement Learning (2024)0.00
• Learning Self-imitating Diverse Policies (2018)0.00
• When Does Self-prediction Help? Understanding Auxiliary Tasks In Reinforcement Learning (2024)0.00
• Spectral Decomposition Representation For Reinforcement Learning (2022)0.00
• Model Predictive Control With Self-supervised Representation Learning (2023)0.00
• Predictive Representations: Building Blocks Of Intelligence (2024)8.09