Bridging State And History Representations: Understanding Self-predictive RL

Abstract

Representations are at the core of all deep reinforcement learning (RL) methods for both Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). Many representation learning methods and theoretical frameworks have been developed to understand what constitutes an effective representation. However, the relationships between these methods and the shared properties among them remain unclear. In this paper, we show that many of these seemingly distinct methods and frameworks for state and history abstractions are, in fact, based on a common idea of self-predictive abstraction. Furthermore, we provide theoretical insights into the widely adopted objectives and optimization, such as the stop-gradient technique, in learning self-predictive representations. These findings together yield a minimalist algorithm to learn self-predictive representations for states and histories. We validate our theories by applying our algorithm to standard MDPs, MDPs with dist

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyni2024bridging

Related papers

• Understanding Self-predictive Learning For Reinforcement Learning (2022)0.00
• A Unifying Framework For Action-conditional Self-predictive Reinforcement Learning (2024)0.00
• Data-efficient Reinforcement Learning With Self-predictive Representations (2020)0.00
• Bootstrapped Representations In Reinforcement Learning (2023)0.00
• Provably Efficient Ucb-type Algorithms For Learning Predictive State Representations (2023)0.00
• On Learning History Based Policies For Controlling Markov Decision Processes (2022)0.00
• Learning Markov State Abstractions For Deep Reinforcement Learning (2021)0.00
• An Empirical Study On The Power Of Future Prediction In Partially Observable Environments (2024)0.00