Leveraging Foundation Models For Content-based Image Retrieval In Radiology

Abstract

Content-based image retrieval (CBIR) has the potential to significantly improve diagnostic aid and medical research in radiology. However, current CBIR systems face limitations due to their specialization to certain pathologies, limiting their utility. On the other hand, several vision foundation models have been shown to produce general-purpose visual features. Therefore, in this work, we propose using vision foundation models as powerful and versatile off-the-shelf feature extractors for content-based image retrieval. Our contributions include: (1) benchmarking a diverse set of vision foundation models on an extensive dataset comprising 1.6 million 2D radiological images across four modalities and 161 pathologies; (2) identifying weakly-supervised models, particularly BiomedCLIP, as highly effective, achieving a achieving a P@1 of up to 0.594 (P@3: 0.590, P@5: 0.588, P@10: 0.583), comparable to specialized CBIR systems but without additional training; (3) conducting an in-depth analy

Stats

• citations12
• S2 citations—
• github stars23
• HF likes0
• heat score11.11
• arxiv keydenner2024leveraging

Related papers

• Evaluating Pre-trained Convolutional Neural Networks And Foundation Models As Feature Extractors For Content-based Medical Image Retrieval (2024)10.65
• Content-based Image Retrieval For Multi-class Volumetric Radiology Images: A Benchmark Study (2024)5.24
• Content-based 3D Image Retrieval And A Colbert-inspired Re-ranking For Tumor Flagging And Staging (2025)0.00
• Content Based Image Retrieval (CBIR) In Remote Clinical Diagnosis And Healthcare (2016)9.23
• Wwfedcbmir: World-wide Federated Content-based Medical Image Retrieval (2023)9.03
• CBIDR: A Novel Method For Information Retrieval Combining Image And Data By Means Of TOPSIS Applied To Medical Diagnosis (2024)0.00
• Cross-modality Sub-image Retrieval Using Contrastive Multimodal Image Representations (2022)6.32
• A Clinically Motivated Self-supervised Approach For Content-based Image Retrieval Of CT Liver Images (2022)9.76