Memory Lens: How Much Memory Does An Agent Use?

Abstract

We propose a new method to study the internal memory used by reinforcement learning policies. We estimate the amount of relevant past information by estimating mutual information between behavior histories and the current action of an agent. We perform this estimation in the passive setting, that is, we do not intervene but merely observe the natural behavior of the agent. Moreover, we provide a theoretical justification for our approach by showing that it yields an implementation-independent lower bound on the minimal memory capacity of any agent that implement the observed policy. We demonstrate our approach by estimating the use of memory of DQN policies on concatenated Atari frames, demonstrating sharply different use of memory across 49 games. The study of memory as information that flows from the past to the current action opens avenues to understand and improve successful reinforcement learning algorithms.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydann2016memory

Related papers

• The Act Of Remembering: A Study In Partially Observable Reinforcement Learning (2020)0.00
• Stable Hadamard Memory: Revitalizing Memory-augmented Agents For Reinforcement Learning (2024)0.00
• Augmented Replay Memory In Reinforcement Learning With Continuous Control (2019)5.24
• How Memory Architecture Affects Learning In A Simple POMDP: The Two-hypothesis Testing Problem (2021)0.00
• Influence-aware Memory Architectures For Deep Reinforcement Learning (2019)2.26
• Adamemento: Adaptive Memory-assisted Policy Optimization For Reinforcement Learning (2024)0.00
• Unsupervised Predictive Memory In A Goal-directed Agent (2018)0.00
• Unraveling The Complexity Of Memory In RL Agents: An Approach For Classification And Evaluation (2024)0.00