FineWeb

🍷 FineWeb 15 trillion tokens of the finest data the 🌐 web has to offer What is it? The 🍷 FineWeb dataset consists of more than 18.5T tokens (originally 15T tokens) of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 datatrove

Papers using FineWeb (5)

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language2025FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering2025The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale2024 · 19 citesProgramming Every Example: Lifting Pre-training Data Quality like Experts at Scale2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale2024