The Pile

An 825 GB diverse open text corpus (22 sources) for training large language models.

Papers using The Pile (4)

Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning2026Chinesewebtext 2.0: Large-scale High-quality Chinese Web Text With Multi-dimensional And Fine-grained Information2024 · 2 citesLate-to-Early Training: LET LLMs Learn Earlier, So Faster and Better2026UniPool: A Globally Shared Expert Pool for Mixture-of-Experts2026