OpenWebText

Dataset Card for "openwebtext" Dataset Summary An open-source replication of the WebText dataset from OpenAI, that was used to train GPT-2. This distribution was created by Aaron Gokaslan and Vanya Cohen of Brown University. Supported Tasks and Leaderboards More Information Needed Languages More Information Needed Data

Papers using OpenWebText (5)

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication2026Long-Context Modeling via GSS-Transformer Hybrid Architecture with Learnable Mixing2026GradPower: Powering Gradients for Faster Language Model Pre-Training2025MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Compute-optimal Scaling of Diffusion Language Models2026The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training2025