Benchmark Granularity And Model Robustness For Image-text Retrieval

Abstract

Image-Text Retrieval (ITR) systems are central to multimodal information access, with Vision-Language Models (VLMs) showing strong performance on standard benchmarks. However, these benchmarks predominantly rely on coarse-grained annotations, limiting their ability to reveal how models perform under real-world conditions, where query granularity varies. Motivated by this gap, we examine how dataset granularity and query perturbations affect retrieval performance and robustness across four architecturally diverse VLMs (ALIGN, AltCLIP, CLIP, and GroupViT). Using both standard benchmarks (MS-COCO, Flickr30k) and their fine-grained variants, we show that richer captions consistently enhance retrieval, especially in text-to-image tasks, where we observe an average improvement of 16.23%, compared to 6.44% in image-to-text. To assess robustness, we introduce a taxonomy of perturbations and conduct extensive experiments, revealing that while perturbations typically degrade performance, they ca

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhendriksen2024benchmark

Related papers

• Benchmarking Robustness Of Text-image Composed Retrieval (2023)2.23
• Fico-itr: Bridging Fine-grained And Coarse-grained Image-text Retrieval For Comparative Performance Analysis (2024)3.58
• Rethinking Benchmarks For Cross-modal Image-text Retrieval (2023)13.11
• Object-aware Query Perturbation For Cross-modal Image-text Retrieval (2024)6.52
• Where Does The Performance Improvement Come From? -- A Reproducibility Concern About Image-text Retrieval (2022)3.36
• A Little More Like This: Text-to-image Retrieval With Vision-language Models Using Relevance Feedback (2025)0.00
• Lexlip: Lexicon-bottlenecked Language-image Pre-training For Large-scale Image-text Retrieval (2023)10.85
• ELIP: Enhanced Visual-language Foundation Models For Image Retrieval (2025)2.26