Selip: Similarity Enhanced Contrastive Language Image Pretraining For Multi-modal Head MRI

Abstract

Despite that deep learning (DL) methods have presented tremendous potential in many medical image analysis tasks, the practical applications of medical DL models are limited due to the lack of enough data samples with manual annotations. By noting that the clinical radiology examinations are associated with radiology reports that describe the images, we propose to develop a foundation model for multi-model head MRI by using contrastive learning on the images and the corresponding radiology findings. In particular, a contrastive learning framework is proposed, where a mixed syntax and semantic similarity matching metric is integrated to reduce the thirst of extreme large dataset in conventional contrastive learning framework. Our proposed similarity enhanced contrastive language image pretraining (SeLIP) is able to effectively extract more useful features. Experiments revealed that our proposed SeLIP performs well in many downstream tasks including image-text retrieval task, classificat

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyliu2025selip

Related papers

• Medclip: Contrastive Learning From Unpaired Medical Images And Text (2022)26.02
• Multi-task Cross-modal Learning For Chest X-ray Image Retrieval (2026)0.00
• Multi-level CLS Token Fusion For Contrastive Learning In Endoscopy Image Classification (2025)0.00
• Learning To Read Where To Look: Disease-aware Vision-language Pretraining For 3D CT (2026)0.00
• More: Multi-modal Contrastive Pre-training With Transformers On X-rays, Ecgs, And Diagnostic Report (2024)0.00
• Dreamlip: Language-image Pre-training With Long Captions (2024)10.61
• SILC: Improving Vision Language Pretraining With Self-distillation (2023)10.21
• Advancing Myopia To Holism: Fully Contrastive Language-image Pre-training (2024)0.00