Towards Mental Time Travel: A Hierarchical Memory For Reinforcement Learning Agents

Abstract

Reinforcement learning agents often forget details of the past, especially after delays or distractor tasks. Agents with common memory architectures struggle to recall and integrate across multiple timesteps of a past event, or even to recall the details of a single timestep that is followed by distractor tasks. To address these limitations, we propose a Hierarchical Chunk Attention Memory (HCAM), which helps agents to remember the past in detail. HCAM stores memories by dividing the past into chunks, and recalls by first performing high-level attention over coarse summaries of the chunks, and then performing detailed attention within only the most relevant chunks. An agent with HCAM can therefore "mentally time-travel" -- remember past events in detail without attending to all intervening events. We show that agents with HCAM substantially outperform agents with other memory architectures at tasks requiring long-term recall, retention, or reasoning over memory. These include recalling

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylampinen2021towards

Related papers

• Stable Hadamard Memory: Revitalizing Memory-augmented Agents For Reinforcement Learning (2024)0.00
• Map-based Experience Replay: A Memory-efficient Solution To Catastrophic Forgetting In Reinforcement Learning (2023)9.23
• Augmented Replay Memory In Reinforcement Learning With Continuous Control (2019)5.24
• Unsupervised Predictive Memory In A Goal-directed Agent (2018)0.00
• Deep Reinforcement Learning With Time-scale Invariant Memory (2024)0.00
• Influence-aware Memory Architectures For Deep Reinforcement Learning (2019)2.26
• Low-pass Recurrent Neural Networks - A Memory Architecture For Longer-term Correlation Discovery (2018)0.00
• HTMRL: Biologically Plausible Reinforcement Learning With Hierarchical Temporal Memory (2020)0.00