An Empirical Study On The Power Of Future Prediction In Partially Observable Environments

Abstract

Learning good representations of historical contexts is one of the core challenges of reinforcement learning (RL) in partially observable environments. While self-predictive auxiliary tasks have been shown to improve performance in fully observed settings, their role in partial observability remains underexplored. In this empirical study, we examine the effectiveness of self-predictive representation learning via future prediction, i.e., predicting next-step observations as an auxiliary task for learning history representations, especially in environments with long-term dependencies. We test the hypothesis that future prediction alone can produce representations that enable strong RL performance. To evaluate this, we introduce \(\texttt\{DRL\}^2\), an approach that explicitly decouples representation learning from reinforcement learning, and compare this approach to end-to-end training across multiple benchmarks requiring long-term memory. Our findings provide evidence that this hypoth

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykwon2024an

Related papers

• Data-efficient Reinforcement Learning With Self-predictive Representations (2020)0.00
• When Does Self-prediction Help? Understanding Auxiliary Tasks In Reinforcement Learning (2024)0.00
• Bridging State And History Representations: Understanding Self-predictive RL (2024)0.00
• Ego-foresight: Self-supervised Learning Of Agent-aware Representations For Improved RL (2024)0.00
• Predicting Future Actions Of Reinforcement Learning Agents (2024)3.58
• A Unifying Framework For Action-conditional Self-predictive Reinforcement Learning (2024)0.00
• Improving Reinforcement Learning Efficiency With Auxiliary Tasks In Non-visual Environments: A Comparison (2023)2.26
• Learning Interpretable Policies In Hindsight-observable Pomdps Through Partially Supervised Reinforcement Learning (2024)2.26