The Act Of Remembering: A Study In Partially Observable Reinforcement Learning

Abstract

Reinforcement Learning (RL) agents typically learn memoryless policies---policies that only consider the last observation when selecting actions. Learning memoryless policies is efficient and optimal in fully observable environments. However, some form of memory is necessary when RL agents are faced with partial observability. In this paper, we study a lightweight approach to tackle partial observability in RL. We provide the agent with an external memory and additional actions to control what, if anything, is written to the memory. At every step, the current memory state is part of the agent's observation, and the agent selects a tuple of actions: one action that modifies the environment and another that modifies the memory. When the external memory is sufficiently expressive, optimal memoryless policies yield globally optimal solutions. Unfortunately, previous attempts to use external memory in the form of binary memory have produced poor results in practice. Here, we investigate alt

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyicarte2020the

Related papers

• Reinforcement Learning Under Partial Observability Guided By Learned Environment Models (2022)6.34
• Stable Hadamard Memory: Revitalizing Memory-augmented Agents For Reinforcement Learning (2024)0.00
• Benchmarking Partial Observability In Reinforcement Learning With A Suite Of Memory-improvable Domains (2025)0.00
• Learning Reward Machines: A Study In Partially Observable Reinforcement Learning (2021)0.00
• Adamemento: Adaptive Memory-assisted Policy Optimization For Reinforcement Learning (2024)0.00
• Active Inference And Reinforcement Learning: A Unified Inference On Continuous State And Action Spaces Under Partial Observability (2022)5.84
• Unsupervised Predictive Memory In A Goal-directed Agent (2018)0.00
• How Memory Architecture Affects Learning In A Simple POMDP: The Two-hypothesis Testing Problem (2021)0.00