Bridging The Training-inference Gap For Dense Phrase Retrieval

Abstract

Building dense retrievers requires a series of standard procedures, including training and validating neural models and creating indexes for efficient search. However, these procedures are often misaligned in that training objectives do not exactly reflect the retrieval scenario at inference time. In this paper, we explore how the gap between training and inference in dense retrieval can be reduced, focusing on dense phrase retrieval (Lee et al., 2021) where billions of representations are indexed at inference. Since validating every dense retriever with a large-scale index is practically infeasible, we propose an efficient way of validating dense retrievers using a small subset of the entire corpus. This allows us to validate various training strategies including unifying contrastive loss terms and using hard negatives for phrase retrieval, which largely reduces the training-inference discrepancy. As a result, we improve top-1 phrase retrieval accuracy by 2~3 points and top-20 passage

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keykim2022bridging

Related papers

• Learning To Retrieve: How To Train A Dense Retrieval Model Effectively And Efficiently (2020)0.00
• Pre-training Vs. Fine-tuning: A Reproducibility Study On Dense Retrieval Knowledge Acquisition (2025)0.95
• Typos-aware Bottlenecked Pre-training For Robust Dense Retrieval (2023)5.84
• Efficiently Teaching An Effective Dense Retriever With Balanced Topic Aware Sampling (2021)17.07
• Dense Passage Retrieval: Is It Retrieving? (2024)6.34
• Unsupervised Dense Information Retrieval With Contrastive Learning (2021)0.00
• Aggretriever: A Simple Approach To Aggregate Textual Representations For Robust Dense Passage Retrieval (2022)13.22
• How To Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval (2023)11.39