M3retrieve: Benchmarking Multimodal Retrieval For Medicine

Abstract

With the increasing use of RetrievalAugmented Generation (RAG), strong retrieval models have become more important than ever. In healthcare, multimodal retrieval models that combine information from both text and images offer major advantages for many downstream tasks such as question answering, cross-modal retrieval, and multimodal summarization, since medical data often includes both formats. However, there is currently no standard benchmark to evaluate how well these models perform in medical settings. To address this gap, we introduce M3Retrieve, a Multimodal Medical Retrieval Benchmark. M3Retrieve, spans 5 domains,16 medical fields, and 4 distinct tasks, with over 1.2 Million text documents and 164K multimodal queries, all collected under approved licenses. We evaluate leading multimodal retrieval models on this benchmark to explore the challenges specific to different medical specialities and to understand their impact on retrieval performance. By releasing M3Retrieve, we aim to

Stats

• citations0
• S2 citations—
• github stars11
• HF likes0
• heat score2.16
• arxiv keyacharya2025m3retrieve

Related papers

• Mmdocir: Benchmarking Multimodal Retrieval For Long Documents (2025)3.58
• MRMR: A Realistic And Expert-level Multidisciplinary Benchmark For Reasoning-intensive Multimodal Retrieval (2025)0.00
• MM-BRIGHT: A Multi-task Multimodal Benchmark For Reasoning-intensive Retrieval (2026)2.60
• R2MED: A Benchmark For Reasoning-driven Medical Retrieval (2025)2.51
• Mrag-bench: Vision-centric Evaluation For Retrieval-augmented Multimodal Models (2024)0.00
• REAL-MM-RAG: A Real-world Multi-modal Retrieval Benchmark (2025)4.52
• Beyond Global Similarity: Towards Fine-grained, Multi-condition Multimodal Retrieval (2026)2.20
• M3DR: Towards Universal Multilingual Multimodal Document Retrieval (2025)0.00