HAL: Improved Text-image Matching By Mitigating Visual Semantic Hubs

Abstract

The hubness problem widely exists in high-dimensional embedding space and is a fundamental source of error for cross-modal matching tasks. In this work, we study the emergence of hubs in Visual Semantic Embeddings (VSE) with application to text-image matching. We analyze the pros and cons of two widely adopted optimization objectives for training VSE and propose a novel hubness-aware loss function (HAL) that addresses previous methods' defects. Unlike (Faghri et al.2018) which simply takes the hardest sample within a mini-batch, HAL takes all samples into account, using both local and global statistics to scale up the weights of "hubs". We experiment our method with various configurations of model architectures and datasets. The method exhibits exceptionally good robustness and brings consistent improvement on the task of text-image matching across all settings. Specifically, under the same model architectures as (Faghri et al. 2018) and (Lee at al. 2018), by switching only the learnin

Stats

• citations24
• S2 citations—
• github stars0
• HF likes0
• heat score10.48
• arxiv keyliu2019hal

Related papers

• Hubness Reduction Improves Sentence-bert Semantic Spaces (2023)0.00
• Adversarial Hubness In Multi-modal Retrieval (2024)0.00
• Neighborretr: Balancing Hub Centrality In Cross-modal Retrieval (2025)4.17
• Improving Visual-semantic Embeddings By Learning Semantically-enhanced Hard Negatives For Cross-modal Information Retrieval (2022)9.41
• Dynamic Visual Semantic Sub-embeddings And Fast Re-ranking (2023)0.00
• Hubness Reduction With Dual Bank Sinkhorn Normalization For Cross-modal Retrieval (2025)0.95
• Image-text Retrieval Via Preserving Main Semantics Of Vision (2023)10.22
• VSE++: Improving Visual-semantic Embeddings With Hard Negatives (2017)0.00