Negative Sample Is Negative In Its Own Way: Tailoring Negative Sentences For Image-text Retrieval

Abstract

Matching model is essential for Image-Text Retrieval framework. Existing research usually train the model with a triplet loss and explore various strategy to retrieve hard negative sentences in the dataset. We argue that current retrieval-based negative sample construction approach is limited in the scale of the dataset thus fail to identify negative sample of high difficulty for every image. We propose our TAiloring neGative Sentences with Discrimination and Correction (TAGS-DC) to generate synthetic sentences automatically as negative samples. TAGS-DC is composed of masking and refilling to generate synthetic negative sentences with higher difficulty. To keep the difficulty during training, we mutually improve the retrieval and generation through parameter sharing. To further utilize fine-grained semantic of mismatch in the negative sentence, we propose two auxiliary tasks, namely word discrimination and word correction to improve the training. In experiments, we verify the effective

Stats

• citations1
• S2 citations—
• github stars5
• HF likes0
• heat score3.81
• arxiv keyfan2021negative

Related papers

• Your Negative May Not Be True Negative: Boosting Image-text Matching With False Negative Elimination (2023)14.32
• Nv-retriever: Improving Text Embedding Models With Effective Hard-negative Mining (2024)0.00
• Sample-specific Debiasing For Better Image-text Models (2023)0.00
• Syneg: Llm-driven Synthetic Hard-negatives For Dense Retrieval (2024)0.00
• Optimizing Dense Retrieval Model Training With Hard Negatives (2021)16.34
• Enhancing Retrieval Performance: An Ensemble Approach For Hard Negative Mining (2024)0.00
• Improving Composed Image Retrieval Via Contrastive Learning With Scaling Positives And Negatives (2024)11.30
• Approximate Nearest Neighbor Negative Contrastive Learning For Dense Text Retrieval (2020)0.00