Pinpoint: Evaluation Of Composed Image Retrieval With Explicit Negatives, Multi-image Queries, And Paraphrase Testing

Abstract

Composed Image Retrieval (CIR) has made significant progress, yet current benchmarks are limited to single ground-truth answers and lack the annotations needed to evaluate false positive avoidance, robustness and multi-image reasoning. We present PinPoint, a comprehensive real world benchmark with 7,635 queries and 329K relevance judgments across 23 query categories. PinPoint advances the field by providing: (1) multiple correct answers (averaging 9.1 per query) (2) explicit hard negatives, (3) six instruction paraphrases per query for robustness testing, (4) multi-image composition support (13.4% of queries), and (5) demographic metadata for fairness evaluation. Based on our analysis of 20+ methods across 4 different major paradigms, we uncover three significant drawbacks: The best methods while achieving mAP@10 of 28.5%, still retrieves irrelevant results (hard negatives) 9% of the time. The best models also exhibit 25.1% performance variation across paraphrases, indicating significa

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymahadev2026pinpoint

Related papers

• A Sanity Check On Composed Image Retrieval (2026)0.00
• Qure: Query-relevant Retrieval Through Hard Negative Sampling In Composed Image Retrieval (2025)2.35
• Rethinking Composed Image Retrieval Evaluation: A Fine-grained Benchmark From Image Editing (2026)0.00
• Instance-level Composed Image Retrieval (2025)0.00
• HINT: Composed Image Retrieval With Dual-path Compositional Contextualized Network (2026)0.78
• Beyond Semantic Search: Towards Referential Anchoring In Composed Image Retrieval (2026)0.00
• Infocir: Multimedia Analysis For Composed Image Retrieval (2026)1.24
• Improving Composed Image Retrieval Via Contrastive Learning With Scaling Positives And Negatives (2024)11.30