Integrating Episodic Memory Into A Reinforcement Learning Agent Using Reservoir Sampling

Abstract

Episodic memory is a psychology term which refers to the ability to recall specific events from the past. We suggest one advantage of this particular type of memory is the ability to easily assign credit to a specific state when remembered information is found to be useful. Inspired by this idea, and the increasing popularity of external memory mechanisms to handle long-term dependencies in deep learning systems, we propose a novel algorithm which uses a reservoir sampling procedure to maintain an external memory consisting of a fixed number of past states. The algorithm allows a deep reinforcement learning agent to learn online to preferentially remember those states which are found to be useful to recall later on. Critically this method allows for efficient online computation of gradient estimates with respect to the write process of the external memory. Thus unlike most prior mechanisms for external memory it is feasible to use in an online reinforcement learning setting.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyoung2018integrating

Related papers

• Sequential Memory Improves Sample And Memory Efficiency In Episodic Control (2021)0.00
• Memory-efficient Episodic Control Reinforcement Learning With Dynamic Online K-means (2019)0.00
• Generating Explanations From Deep Reinforcement Learning Using Episodic Memory (2022)0.00
• Towards Sample-efficient Episodic Control With DAC-ML (2020)0.00
• Reservoir Computing For Fast, Simplified Reinforcement Learning On Memory Tasks (2024)0.00
• Augmented Replay Memory In Reinforcement Learning With Continuous Control (2019)5.24
• Generalization Of Reinforcement Learners With Working And Episodic Memory (2019)0.00
• Episodic Reinforcement Learning With Expanded State-reward Space (2024)0.00