Emergent Temporal Abstractions In Autoregressive Models Enable Hierarchical Reinforcement Learning
2025 · Seijin Kobayashi, Yanick Schimpf, Maximilian Schlegel, et al.
Abstract
Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. During RL, these models explore by generating new outputs, one token at a time. However, sampling actions token-by-token can result in highly inefficient learning, particularly when rewards are sparse. Here, we show that it is possible to overcome this problem by acting and exploring within the internal representations of an autoregressive model. Specifically, to discover temporally-abstract actions, we introduce a higher-order, non-causal sequence model whose outputs control the residual stream activations of a base autoregressive model. On grid world and MuJoCo-based tasks with hierarchical structure, we find that the higher-order model learns to compress long activation sequence chunks onto internal controllers. Critically, each controller executes a sequence of behaviorally meaningful actions that unfold ov
Authors
(none)
Tags
Stats
Related papers
- Learning Representations In Model-free Hierarchical Reinforcement Learning (2018)11.49
- Self-organization Of Action Hierarchy And Compositionality By Reinforcement Learning With Recurrent Neural Networks (2019)8.60
- Multi-horizon Representations With Hierarchical Forward Models For Reinforcement Learning (2022)0.00
- Hierarchical Deep Multiagent Reinforcement Learning With Temporal Abstraction (2018)0.00
- Autonomous Option Invention For Continual Hierarchical Reinforcement Learning And Planning (2024)2.26
- Exploring The Limits Of Hierarchical World Models In Reinforcement Learning (2024)6.34
- HTMRL: Biologically Plausible Reinforcement Learning With Hierarchical Temporal Memory (2020)0.00
- Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction And Intrinsic Motivation (2016)0.00