Learning To Retrieve Passages Without Supervision

Abstract

Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval. Our "recurring span retrieval" approach uses recurring spans across passages in a document to create pseudo examples for contrastive learning. Our pretraining scheme directly controls for term overlap across pseudo queries and relevant passages, thus allowing to model both lexical and semantic relations between them. The resulting model, named Spider, performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Moreo

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keyram2021learning

Related papers

• Synthetic Target Domain Supervision For Open Retrieval QA (2022)4.52
• Pre-training Multi-modal Dense Retrievers For Outside-knowledge Visual Question Answering (2023)7.50
• Query-as-context Pre-training For Dense Passage Retrieval (2022)7.68
• Questions Are All You Need To Train A Dense Passage Retriever (2022)10.35
• Laprador: Unsupervised Pretrained Dense Retriever For Zero-shot Text Retrieval (2022)8.82
• Domain-matched Pre-training Tasks For Dense Retrieval (2021)5.24
• Revela: Dense Retriever Learning Via Language Modeling (2025)0.00
• How To Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval (2023)11.39