Query-as-context Pre-training For Dense Passage Retrieval
2022 · Xing Wu, Guangyuan Ma, Wanhui Qian, et al.
Abstract
Recently, methods have been developed to improve the performance of dense passage retrieval by using context-supervised pre-training. These methods simply consider two passages from the same document to be relevant, without taking into account the possibility of weakly correlated pairs. Thus, this paper proposes query-as-context pre-training, a simple yet effective pre-training technique to alleviate the issue. Query-as-context pre-training assumes that the query derived from a passage is more likely to be relevant to that passage and forms a passage-query pair. These passage-query pairs are then used in contrastive or generative context-supervised pre-training. The pre-trained models are evaluated on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks. Experimental results show that query-as-context pre-training brings considerable gains and meanwhile speeds up training, demonstrating its effectiveness and efficiency. Our code will be available at https://g
Authors
(none)
Tags
Stats
Related papers
- Unsupervised Context Aware Sentence Representation Pretraining For Multi-lingual Dense Retrieval (2022)3.58
- Cot-mote: Exploring Contextual Masked Auto-encoder Pre-training With Mixture-of-textual-experts For Passage Retrieval (2023)0.00
- DAPR: A Benchmark On Document-aware Passage Retrieval (2023)5.18
- Learning To Retrieve Passages Without Supervision (2021)8.09
- Contextualized Query Embeddings For Conversational Search (2021)10.21
- Augmenting Passage Representations With Query Generation For Enhanced Cross-lingual Dense Retrieval (2023)8.14
- Simlm: Pre-training With Representation Bottleneck For Dense Passage Retrieval (2022)20.27
- Cot-mae V2: Contextual Masked Auto-encoder With Multi-view Modeling For Passage Retrieval (2023)0.00