Your Negative May Not Be True Negative: Boosting Image-text Matching With False Negative Elimination

Abstract

Most existing image-text matching methods adopt triplet loss as the optimization objective, and choosing a proper negative sample for the triplet of <anchor, positive, negative> is important for effectively training the model, e.g., hard negatives make the model learn efficiently and effectively. However, we observe that existing methods mainly employ the most similar samples as hard negatives, which may not be true negatives. In other words, the samples with high similarity but not paired with the anchor may reserve positive semantic associations, and we call them false negatives. Repelling these false negatives in triplet loss would mislead the semantic representation learning and result in inferior retrieval performance. In this paper, we propose a novel False Negative Elimination (FNE) strategy to select negatives via sampling, which could alleviate the problem introduced by false negatives. Specifically, we first construct the distributions of positive and negative samples separat

Stats

• citations35
• S2 citations—
• github stars20
• HF likes0
• heat score14.32
• arxiv keyli2023your

Related papers

• Negative Sample Is Negative In Its Own Way: Tailoring Negative Sentences For Image-text Retrieval (2021)3.81
• Hard Negative Examples Are Hard, But Useful (2020)14.76
• Sample-specific Debiasing For Better Image-text Models (2023)0.00
• Boosting Weak Positives For Text Based Person Search (2025)0.00
• Nv-retriever: Improving Text Embedding Models With Effective Hard-negative Mining (2024)0.00
• Deep Boosting Learning: A Brand-new Cooperative Approach For Image-text Matching (2024)9.73
• No Captions, No Problem: Captionless 3D-CLIP Alignment With Hard Negatives Via CLIP Knowledge And Llms (2024)0.00
• Gistembed: Guided In-sample Selection Of Training Negatives For Text Embedding Fine-tuning (2024)0.00