FineWeb

🍷 FineWeb 15 trillion tokens of the finest data the 🌐 web has to offer What is it? The 🍷 FineWeb dataset consists of more than 18.5T tokens (originally 15T tokens) of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 datatrove

Papers using FineWeb (6)

Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models2026Memorization Dynamics in Knowledge Distillation for Language Models2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration2026FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language2025HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation2025