INQUIRE: A Natural World Text-to-image Retrieval Benchmark

Abstract

We introduce INQUIRE, a text-to-image retrieval benchmark designed to challenge multimodal vision-language models on expert-level queries. INQUIRE includes iNaturalist 2024 (iNat24), a new dataset of five million natural world images, along with 250 expert-level retrieval queries. These queries are paired with all relevant images comprehensively labeled within iNat24, comprising 33,000 total matches. Queries span categories such as species identification, context, behavior, and appearance, emphasizing tasks that require nuanced image understanding and domain expertise. Our benchmark evaluates two core retrieval tasks: (1) INQUIRE-Fullrank, a full dataset ranking task, and (2) INQUIRE-Rerank, a reranking task for refining top-100 retrievals. Detailed evaluation of a range of recent multimodal models demonstrates that INQUIRE poses a significant challenge, with the best models failing to achieve an mAP@50 above 50%. In addition, we show that reranking with more powerful multimodal models

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyvendrow2024inquire

Related papers

• MM-BRIGHT: A Multi-task Multimodal Benchmark For Reasoning-intensive Retrieval (2026)2.60
• IDMR: Towards Instance-driven Precise Visual Correspondence In Multimodal Retrieval (2025)2.29
• Benchmark Granularity And Model Robustness For Image-text Retrieval (2024)0.00
• Visr-bench: An Empirical Study On Visual Retrieval-augmented Generation For Multilingual Long Document Understanding (2025)0.00
• Deepimagesearch: Benchmarking Multimodal Agents For Context-aware Image Retrieval In Visual Histories (2026)0.00
• Uniir: Training And Benchmarking Universal Multimodal Information Retrievers (2023)10.48
• End-to-end Knowledge Retrieval With Multi-modal Queries (2023)8.35
• Few Shots Text To Image Retrieval: New Benchmarking Dataset And Optimization Methods (2026)0.00