Transformers Are Sample-efficient World Models

Abstract

Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have been designed to address this issue, with learning in the imagination of a world model being one of the most prominent approaches. However, while virtually unlimited interaction with a simulated environment sounds appealing, the world model has to be accurate over extended periods of time. Motivated by the success of Transformers in sequence modeling tasks, we introduce IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer. With the equivalent of only two hours of gameplay in the Atari 100k benchmark, IRIS achieves a mean human normalized score of 1.046, and outperforms humans on 10 out of 26 games, setting a new state of the art for methods without lookahead search. To foster future research on Transformers and world models for sample-ef

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymicheli2022transformers

Related papers

• Decentralized Transformers With Centralized Aggregation Are Sample-efficient Multi-agent World Models (2024)0.00
• Learning To Play Atari In A World Of Tokens (2024)0.00
• STORM: Efficient Stochastic Transformer Based World Models For Reinforcement Learning (2023)4.52
• Object-centric World Models For Causality-aware Reinforcement Learning (2025)0.00
• Smaller World Models For Reinforcement Learning (2020)0.00
• From Memories To Maps: Mechanisms Of In-context Reinforcement Learning In Transformers (2025)0.00
• Hieros: Hierarchical Imagination On Structured State Space Sequence World Models (2023)0.00
• Do Transformer World Models Give Better Policy Gradients? (2024)0.00