Data-efficient Ranking Distillation For Image Retrieval

Abstract

Recent advances in deep learning has lead to rapid developments in the field of image retrieval. However, the best performing architectures incur significant computational cost. Recent approaches tackle this issue using knowledge distillation to transfer knowledge from a deeper and heavier architecture to a much smaller network. In this paper we address knowledge distillation for metric learning problems. Unlike previous approaches, our proposed method jointly addresses the following constraints i) limited queries to teacher model, ii) black box teacher model with access to the final output representation, and iii) small fraction of original training data without any ground-truth labels. In addition, the distillation method does not require the student and teacher to have same dimensionality. Addressing these constraints reduces computation requirements, dependency on large-scale training datasets and addresses practical scenarios of limited or partial access to private data such as te

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylaskar2020data

Related papers

• Context Unaware Knowledge Distillation For Image Retrieval (2022)0.60
• Towards A Smaller Student: Capacity Dynamic Distillation For Efficient Image Retrieval (2023)10.07
• Embeddistill: A Geometric Knowledge Distillation For Information Retrieval (2023)0.00
• Knowledge Distillation In Document Retrieval (2019)0.00
• Learning Metrics From Teachers: Compact Networks For Image Embedding (2019)18.74
• Large-to-small Image Resolution Asymmetry In Deep Metric Learning (2022)9.62
• Let All Be Whitened: Multi-teacher Distillation For Efficient Visual Retrieval (2023)8.86
• Curriculum Learning For Dense Retrieval Distillation (2022)11.49