S2SD: Simultaneous Similarity-based Self-distillation For Deep Metric Learning

Abstract

Deep Metric Learning (DML) provides a crucial tool for visual similarity and zero-shot applications by learning generalizing embedding spaces, although recent work in DML has shown strong performance saturation across training objectives. However, generalization capacity is known to scale with the embedding space dimensionality. Unfortunately, high dimensional embeddings also create higher retrieval cost for downstream applications. To remedy this, we propose \emph\{Simultaneous Similarity-based Self-distillation (S2SD). S2SD extends DML with knowledge distillation from auxiliary, high-dimensional embedding and feature spaces to leverage complementary context during training while retaining test-time cost and with negligible changes to the training time. Experiments and ablations across different objectives and standard benchmarks show S2SD offers notable improvements of up to 7% in Recall@1, while also setting a new state-of-the-art. Code available at https://github.com/MLforHealth/S2

Stats

• citations0
• S2 citations—
• github stars44
• HF likes0
• heat score3.31
• arxiv keyroth2020s2sd

Code

• MLforHealth/S2—

Related papers

• Guided Deep Metric Learning (2022)6.77
• Revisiting Training Strategies And Generalization Performance In Deep Metric Learning (2020)5.08
• Improving Deep Metric Learning By Divide And Conquer (2021)8.09
• Deep Metric Learning-based Semi-supervised Regression With Alternate Learning (2022)5.24
• Learning With Memory-based Virtual Classes For Deep Metric Learning (2021)9.92
• DAS: Densely-anchored Sampling For Deep Metric Learning (2022)9.76
• Integrating Language Guidance Into Vision-based Deep Metric Learning (2022)14.04
• Ensemble Of Loss Functions To Improve Generalizability Of Deep Metric Learning Methods (2021)7.16