Benchmarking Robustness Of Contrastive Learning Models For Medical Image-report Retrieval

Abstract

Medical images and reports offer invaluable insights into patient health. The heterogeneity and complexity of these data hinder effective analysis. To bridge this gap, we investigate contrastive learning models for cross-domain retrieval, which associates medical images with their corresponding clinical reports. This study benchmarks the robustness of four state-of-the-art contrastive learning models: CLIP, CXR-RePaiR, MedCLIP, and CXR-CLIP. We introduce an occlusion retrieval task to evaluate model performance under varying levels of image corruption. Our findings reveal that all evaluated models are highly sensitive to out-of-distribution data, as evidenced by the proportional decrease in performance with increasing occlusion levels. While MedCLIP exhibits slightly more robustness, its overall performance remains significantly behind CXR-CLIP and CXR-RePaiR. CLIP, trained on a general-purpose dataset, struggles with medical image-report retrieval, highlighting the importance of domai

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydeanda2025benchmarking

Related papers

• Masked Contrastive Reconstruction For Cross-modal Medical Image-report Retrieval (2023)0.00
• Multi-task Cross-modal Learning For Chest X-ray Image Retrieval (2026)0.00
• Benchmarking Vision-language Contrastive Methods For Medical Representation Learning (2024)0.00
• Medclip: Contrastive Learning From Unpaired Medical Images And Text (2022)26.02
• Prototype-enhanced Confidence Modeling For Cross-modal Medical Image-report Retrieval (2025)0.00
• Radir: A Scalable Framework For Multi-grained Medical Image Retrieval Via Radiology Report Mining (2025)0.00
• Medprobclip: Probabilistic Adaptation Of Vision-language Foundation Model For Reliable Radiograph-report Retrieval (2026)0.00
• Evaluating Contrastive Models For Instance-based Image Retrieval (2021)5.24