Simlm: Pre-training With Representation Bottleneck For Dense Passage Retrieval

Abstract

In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost. Our code and model check points are available at https://github.com/microsoft/unilm/tree/mast

Stats

• citations34
• S2 citations—
• github stars22114
• HF likes0
• heat score20.27
• arxiv keywang2022simlm

Code

• microsoft/unilm—

Related papers

• Unsupervised Context Aware Sentence Representation Pretraining For Multi-lingual Dense Retrieval (2022)3.58
• Query-as-context Pre-training For Dense Passage Retrieval (2022)7.68
• Condenser: A Pre-training Architecture For Dense Retrieval (2021)14.90
• Investigating Multi-layer Representations For Dense Passage Retrieval (2025)0.00
• Modeling Sequential Sentence Relation To Improve Cross-lingual Dense Retrieval (2023)1.20
• I^3 Retriever: Incorporating Implicit Interaction In Pre-trained Language Models For Passage Retrieval (2023)7.16
• CSPLADE: Learned Sparse Retrieval With Causal Language Models (2025)0.00
• SLQ: Bridging Modalities Via Shared Latent Queries For Retrieval With Frozen Mllms (2026)0.00