Rethinking Composed Image Retrieval Evaluation: A Fine-grained Benchmark From Image Editing

Abstract

Composed Image Retrieval (CIR) is a pivotal and complex task in multimodal understanding. Current CIR benchmarks typically feature limited query categories and fail to capture the diverse requirements of real-world scenarios. To bridge this evaluation gap, we leverage image editing to achieve precise control over modification types and content, enabling a pipeline for synthesizing queries across a broad spectrum of categories. Using this pipeline, we construct EDIR, a novel fine-grained CIR benchmark. EDIR encompasses 5,000 high-quality queries structured across five main categories and fifteen subcategories. Our comprehensive evaluation of 13 multimodal embedding models reveals a significant capability gap; even state-of-the-art models (e.g., RzenEmbed and GME) struggle to perform consistently across all subcategories, highlighting the rigorous nature of our benchmark. Through comparative analysis, we further uncover inherent limitations in existing benchmarks, such as modality biases

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysong2026rethinking

Related papers

• A Sanity Check On Composed Image Retrieval (2026)0.00
• Infocir: Multimedia Analysis For Composed Image Retrieval (2026)1.24
• Instance-level Composed Image Retrieval (2025)0.00
• Finecir: Explicit Parsing Of Fine-grained Modification Semantics For Composed Image Retrieval (2025)2.16
• DQE-CIR: Distinctive Query Embeddings Through Learnable Attribute Weights And Target Relative Negative Sampling In Composed Image Retrieval (2026)0.00
• FBCIR: Balancing Cross-modal Focuses In Composed Image Retrieval (2026)0.00
• HINT: Composed Image Retrieval With Dual-path Compositional Contextualized Network (2026)0.78
• OFFSET: Segmentation-based Focus Shift Revision For Composed Image Retrieval (2025)5.84