FineWeb-Edu

📚 FineWeb-Edu 1.3 trillion tokens of the finest educational data the 🌐 web has to offer Paper: https://arxiv.org/abs/2406.17557 What is it? 📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.

Papers using FineWeb-Edu (6)

Co-LMLM: Continuous-Query Limited Memory Language Models2026Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration2026Attention Needs to Focus: A Unified Perspective on Attention Allocation2026FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition2025Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language2024