OpenWebText

Dataset Card for "openwebtext" Dataset Summary An open-source replication of the WebText dataset from OpenAI, that was used to train GPT-2. This distribution was created by Aaron Gokaslan and Vanya Cohen of Brown University. Supported Tasks and Leaderboards More Information Needed Languages More Information Needed Data

Papers using OpenWebText (3)

Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing2026GradPower: Powering Gradients for Faster Language Model Pre-Training2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training2025