Pairdistill: Pairwise Relevance Distillation For Dense Retrieval

Abstract

Effective information retrieval (IR) from vast datasets relies on advanced techniques to extract relevant information in response to queries. Recent advancements in dense retrieval have showcased remarkable efficacy compared to traditional sparse retrieval methods. To further enhance retrieval performance, knowledge distillation techniques, often leveraging robust cross-encoder rerankers, have been extensively explored. However, existing approaches primarily distill knowledge from pointwise rerankers, which assign absolute relevance scores to documents, thus facing challenges related to inconsistent comparisons. This paper introduces Pairwise Relevance Distillation (PairDistill) to leverage pairwise reranking, offering fine-grained distinctions between similarly relevant documents to enrich the training of dense retrieval models. Our experiments demonstrate that PairDistill outperforms existing methods, achieving new state-of-the-art results across multiple benchmarks. This highlights

Stats

• citations3
• S2 citations—
• github stars22
• HF likes0
• heat score7.24
• arxiv keyhuang2024pairdistill

Related papers

• Embeddistill: A Geometric Knowledge Distillation For Information Retrieval (2023)0.00
• Curriculum Learning For Dense Retrieval Distillation (2022)11.49
• PROD: Progressive Distillation For Dense Retrieval (2022)9.23
• Enhancing The Ranking Context Of Dense Retrieval Methods Through Reciprocal Nearest Neighbors (2023)4.52
• Data-efficient Ranking Distillation For Image Retrieval (2020)0.00
• Learning Effective Representations For Retrieval Using Self-distillation With Adaptive Relevance Margins (2024)2.26
• Refit: Relevance Feedback From A Reranker During Inference (2023)0.00
• Translate-distill: Learning Cross-language Dense Retrieval By Translation And Distillation (2024)8.60