Associative Memory Based Experience Replay For Deep Reinforcement Learning

Abstract

Experience replay is an essential component in deep reinforcement learning (DRL), which stores the experiences and generates experiences for the agent to learn in real time. Recently, prioritized experience replay (PER) has been proven to be powerful and widely deployed in DRL agents. However, implementing PER on traditional CPU or GPU architectures incurs significant latency overhead due to its frequent and irregular memory accesses. This paper proposes a hardware-software co-design approach to design an associative memory (AM) based PER, AMPER, with an AM-friendly priority sampling operation. AMPER replaces the widely-used time-costly tree-traversal-based priority sampling in PER while preserving the learning performance. Further, we design an in-memory computing hardware architecture based on AM to support AMPER by leveraging parallel in-memory search operations. AMPER shows comparable learning performance while achieving 55x to 270x latency improvement when running on the proposed

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyli2022associative

Related papers

• Prioritized Sequence Experience Replay (2019)0.00
• A Deeper Look At Experience Replay (2017)0.00
• Remember And Forget For Experience Replay (2018)0.00
• Accmer: Accelerating Multi-agent Experience Replay With Cache Locality-aware Prioritization (2023)5.24
• Map-based Experience Replay: A Memory-efficient Solution To Catastrophic Forgetting In Reinforcement Learning (2023)9.23
• Improving Experience Replay Through Modeling Of Similar Transitions' Sets (2021)0.00
• Introspective Experience Replay: Look Back When Surprised (2022)0.00
• Reward Prediction Error Prioritisation In Experience Replay: The RPE-PER Method (2025)0.00