In-context Pretraining: Language Modeling Beyond Document Boundaries

Abstract

Large language models (LMs) are currently trained to predict tokens given document prefixes, enabling them to directly perform long-form generation and prompting-style tasks which can be reduced to document completion. Existing pretraining pipelines train LMs by concatenating random sets of short documents to create input contexts but the prior documents provide no signal for predicting the next document. We instead present In-Context Pretraining, a new approach where language models are pretrained on a sequence of related documents, thereby explicitly encouraging them to read and reason across document boundaries. We can do In-Context Pretraining by simply changing the document ordering so that each context contains related documents, and directly applying existing pretraining pipelines. However, this document sorting problem is challenging. There are billions of documents and we would like the sort to maximize contextual similarity for every document without repeating any data. To do

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyshi2023in

Related papers

• Unsupervised Context Aware Sentence Representation Pretraining For Multi-lingual Dense Retrieval (2022)3.58
• Temporal Perceiving Video-language Pre-training (2023)0.00
• Query-as-context Pre-training For Dense Passage Retrieval (2022)7.68
• Lotlip: Improving Language-image Pre-training For Long Text Understanding (2024)2.26
• Modeling Sequential Sentence Relation To Improve Cross-lingual Dense Retrieval (2023)1.20
• Compressing Then Matching: An Efficient Pre-training Paradigm For Multimodal Embedding (2025)0.00
• Pre-training Tasks For Embedding-based Large-scale Retrieval (2020)0.00
• Towards In-context Scene Understanding (2023)0.00