Maximal Matching Matters: Preventing Representation Collapse For Robust Cross-modal Retrieval

Abstract

Cross-modal image-text retrieval is challenging because of the diverse possible associations between content from different modalities. Traditional methods learn a single-vector embedding to represent semantics of each sample, but struggle to capture nuanced and diverse relationships that can exist across modalities. Set-based approaches, which represent each sample with multiple embeddings, offer a promising alternative, as they can capture richer and more diverse relationships. In this paper, we show that, despite their promise, these set-based representations continue to face issues including sparse supervision and set collapse, which limits their effectiveness. To address these challenges, we propose Maximal Pair Assignment Similarity to optimize one-to-one matching between embedding sets which preserve semantic diversity within the set. We also introduce two loss functions to further enhance the representations: Global Discriminative Loss to enhance distinction among embeddings, a

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyalomari2025maximal

Related papers

• Preserving Semantic Neighborhoods For Robust Cross-modal Retrieval (2020)10.07
• Learning To Rematch Mismatched Pairs For Robust Cross-modal Retrieval (2024)13.82
• Improving Cross-modal Retrieval With Set Of Diverse Embeddings (2022)13.55
• Multimodal Representation Alignment For Cross-modal Information Retrieval (2025)0.00
• Cross-modal Image Retrieval With Deep Mutual Information Maximization (2021)9.59
• Discriminative Semantic Transitive Consistency For Cross-modal Learning (2021)0.00
• Look, Imagine And Match: Improving Textual-visual Cross-modal Retrieval With Generative Models (2017)18.52
• Adversarial Cross-modal Retrieval Via Learning And Transferring Single-modal Similarities (2019)8.60