Predictive Information Accelerates Learning In RL

Abstract

The Predictive Information is the mutual information between the past and the future, I(X_past; X_future). We hypothesize that capturing the predictive information is useful in RL, since the ability to model what will happen next is necessary for success on many tasks. To test our hypothesis, we train Soft Actor-Critic (SAC) agents from pixels with an auxiliary task that learns a compressed representation of the predictive information of the RL environment dynamics using a contrastive version of the Conditional Entropy Bottleneck (CEB) objective. We refer to these as Predictive Information SAC (PI-SAC) agents. We show that PI-SAC agents can substantially improve sample efficiency over challenging baselines on tasks from the DM Control suite of continuous control environments. We evaluate PI-SAC agents by comparing against uncompressed PI-SAC agents, other compressed and uncompressed agents, and SAC agents directly trained from pixels. Our implementation is given on GitHub.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylee2020predictive

Related papers

• Data-efficient Reinforcement Learning With Self-predictive Representations (2020)0.00
• Boosting Soft Actor-critic: Emphasizing Recent Experience Without Forgetting The Past (2019)0.00
• Context-based Soft Actor Critic For Environments With Non-stationary Dynamics (2021)0.00
• A Unifying Framework For Action-conditional Self-predictive Reinforcement Learning (2024)0.00
• Deep RL With Information Constrained Policies: Generalization In Continuous Control (2020)0.00
• Improved Soft Actor-critic: Mixing Prioritized Off-policy Samples With On-policy Experience (2021)0.00
• Efficient Deep Reinforcement Learning With Predictive Processing Proximal Policy Optimization (2022)0.00
• Discrete And Continuous Action Representation For Practical RL In Video Games (2019)0.00