PROD: Progressive Distillation For Dense Retrieval

Abstract

Knowledge distillation is an effective way to transfer knowledge from a strong teacher to an efficient student model. Ideally, we expect the better the teacher is, the better the student. However, this expectation does not always come true. It is common that a better teacher model results in a bad student via distillation due to the nonnegligible gap between teacher and student. To bridge the gap, we propose PROD, a PROgressive Distillation method, for dense retrieval. PROD consists of a teacher progressive distillation and a data progressive distillation to gradually improve the student. We conduct extensive experiments on five widely-used benchmarks, MS MARCO Passage, TREC Passage 19, TREC Document 19, MS MARCO Document and Natural Questions, where PROD achieves the state-of-the-art within the distillation methods for dense retrieval. The code and models will be released.

Stats

• citations16
• S2 citations—
• github stars0
• HF likes0
• heat score9.23
• arxiv keylin2022prod

Related papers

• Curriculum Learning For Dense Retrieval Distillation (2022)11.49
• Pairdistill: Pairwise Relevance Distillation For Dense Retrieval (2024)7.24
• Embeddistill: A Geometric Knowledge Distillation For Information Retrieval (2023)0.00
• Data-efficient Ranking Distillation For Image Retrieval (2020)0.00
• Knowledge Distillation In Document Retrieval (2019)0.00
• Beyond Hard Negatives: The Importance Of Score Distribution In Knowledge Distillation For Dense Retrieval (2026)0.00
• Teaching Dense Retrieval Models To Specialize With Listwise Distillation And LLM Data Augmentation (2025)0.00
• Dense Passage Retrieval: Is It Retrieving? (2024)6.34