Knowledge Distillation In Document Retrieval

Abstract

Complex deep learning models now achieve state of the art performance for many document retrieval tasks. The best models process the query or claim jointly with the document. However for fast scalable search it is desirable to have document embeddings which are independent of the claim. In this paper we show that knowledge distillation can be used to encourage a model that generates claim independent document encodings to mimic the behavior of a more complex model which generates claim dependent encodings. We explore this approach in document retrieval for a fact extraction and verification task. We show that by using the soft labels from a complex cross attention teacher model, the performance of claim independent student LSTM or CNN models is improved across all the ranking metrics. The student models we use are 12x faster in runtime and 20x smaller in number of parameters than the teacher

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyshakeri2019knowledge

Related papers

• Embeddistill: A Geometric Knowledge Distillation For Information Retrieval (2023)0.00
• Context Unaware Knowledge Distillation For Image Retrieval (2022)0.60
• Data-efficient Ranking Distillation For Image Retrieval (2020)0.00
• Learning Effective Representations For Retrieval Using Self-distillation With Adaptive Relevance Margins (2024)2.26
• Curriculum Learning For Dense Retrieval Distillation (2022)11.49
• Teaching Dense Retrieval Models To Specialize With Listwise Distillation And LLM Data Augmentation (2025)0.00
• Beyond Hard Negatives: The Importance Of Score Distribution In Knowledge Distillation For Dense Retrieval (2026)0.00
• Translate-distill: Learning Cross-language Dense Retrieval By Translation And Distillation (2024)8.60