Sequence Compression Speeds Up Credit Assignment In Reinforcement Learning

Abstract

Temporal credit assignment in reinforcement learning is challenging due to delayed and stochastic outcomes. Monte Carlo targets can bridge long delays between action and consequence but lead to high-variance targets due to stochasticity. Temporal difference (TD) learning uses bootstrapping to overcome variance but introduces a bias that can only be corrected through many iterations. TD(\(\lambda\)) provides a mechanism to navigate this bias-variance tradeoff smoothly. Appropriately selecting \(\lambda\) can significantly improve performance. Here, we propose Chunked-TD, which uses predicted probabilities of transitions from a model for computing \(\lambda\)-return targets. Unlike other model-based solutions to credit assignment, Chunked-TD is less vulnerable to model inaccuracies. Our approach is motivated by the principle of history compression and 'chunks' trajectories for conventional TD learning. Chunking with learned world models compresses near-deterministic regions of the enviro

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyramesh2024sequence

Related papers

• Demystifying The Recency Heuristic In Temporal-difference Learning (2024)0.00
• Selective Credit Assignment (2022)0.00
• Adaptive Pairwise Weights For Temporal Credit Assignment (2021)0.00
• A Survey Of Temporal Credit Assignment In Deep Reinforcement Learning (2023)0.00
• Temporal Difference Learning With Compressed Updates: Error-feedback Meets Reinforcement Learning (2023)0.00
• TD Or Not TD: Analyzing The Role Of Temporal Differencing In Deep Reinforcement Learning (2018)0.00
• Temporal-difference Learning Using Distributed Error Signals (2024)0.00
• Discerning Temporal Difference Learning (2023)0.00