C4

The Colossal Clean Crawled Corpus — a large cleaned web-text corpus used to pretrain T5 and many other LLMs.

Papers using C4 (9)

Chinesewebtext 2.0: Large-scale High-quality Chinese Web Text With Multi-dimensional And Fine-grained Information2024 · 2 citesDLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training2026GradPower: Powering Gradients for Faster Language Model Pre-Training2025Signature filtering: a lightweight enhancement for statistical watermark detection in large language models2026Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columns2026Layer-wise dynamic rank for compressing large language models2025Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models2025SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training2025