Do Lessons From Metric Learning Generalize To Image-caption Retrieval?

Abstract

The triplet loss with semi-hard negatives has become the de facto choice for image-caption retrieval (ICR) methods that are optimized from scratch. Recent progress in metric learning has given rise to new loss functions that outperform the triplet loss on tasks such as image retrieval and representation learning. We ask whether these findings generalize to the setting of ICR by comparing three loss functions on two ICR methods. We answer this question negatively: the triplet loss with semi-hard negative mining still outperforms newly introduced loss functions from metric learning on the ICR task. To gain a better understanding of these outcomes, we introduce an analysis method to compare loss functions by counting how many samples contribute to the gradient w.r.t. the query representation during optimization. We find that loss functions that result in lower evaluation scores on the ICR task, in general, take too many (non-informative) samples into account when computing a gradient w.r.

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keybleeker2022do

Related papers

• Variance & Greediness: A Comparative Study Of Metric-learning Losses (2026)0.00
• Hard Negative Examples Are Hard, But Useful (2020)14.76
• Deep Metric Learning Beyond Binary Supervision (2019)14.39
• Constellation Loss: Improving The Efficiency Of Deep Metric Learning Loss Functions For Optimal Embedding (2019)0.00
• Unbiased Evaluation Of Deep Metric Learning Algorithms (2019)0.00
• Comparing Contrastive And Triplet Loss: Variance Analysis And Optimization Behavior (2025)0.00
• Three Things To Know About Deep Metric Learning (2024)0.00
• No Captions, No Problem: Captionless 3D-CLIP Alignment With Hard Negatives Via CLIP Knowledge And Llms (2024)0.00