From Memories To Maps: Mechanisms Of In-context Reinforcement Learning In Transformers

Abstract

Humans and animals show remarkable learning efficiency, adapting to new environments with minimal experience. This capability is not well captured by standard reinforcement learning algorithms that rely on incremental value updates. Rapid adaptation likely depends on episodic memory -- the ability to retrieve specific past experiences to guide decisions in novel contexts. Transformers provide a useful setting for studying these questions because of their ability to learn rapidly in-context and because their key-value architecture resembles episodic memory systems in the brain. We train a transformer to in-context reinforcement learn in a distribution of planning tasks inspired by rodent behavior. We then characterize the learning algorithms that emerge in the model. We first find that representation learning is supported by in-context structure learning and cross-context alignment, where representations are aligned across environments with different sensory stimuli. We next demonstrate

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfang2025from

Related papers

• Transformers Can Learn Temporal Difference Methods For In-context Reinforcement Learning (2024)0.00
• Recurrent Action Transformer With Memory (2023)0.00
• Learning To Play Atari In A World Of Tokens (2024)0.00
• Cross-episodic Curriculum For Transformer Agents (2023)0.00
• Transformers Are Sample-efficient World Models (2022)0.00
• Transformers As Game Players: Provable In-context Game-playing Capabilities Of Pre-trained Models (2024)0.00
• Energy-efficient Deep Reinforcement Learning With Spiking Transformers (2025)2.26
• Transformer Based Reinforcement Learning For Games (2019)0.00