Multi-task Cross-modal Learning For Chest X-ray Image Retrieval

Abstract

CLIP and BiomedCLIP are examples of vision-language foundation models and offer strong cross-modal embeddings; however, they are not optimized for fine-grained medical retrieval tasks, such as retrieving clinically relevant radiology reports using chest X-ray (CXR) image queries. To address this shortcoming, we propose a multi-task learning framework to fine-tune BiomedCLIP and evaluate improvements to CXR image-text retrieval. Using BiomedCLIP as the backbone, we incorporate a lightweight MLP projector head trained with a multi-task composite loss function that includes: (1) a binary cross-entropy loss to distinguish normal from abnormal CXR studies, (2) a supervised contrastive loss to reinforce intra-class consistency, and (3) a CLIP loss to maintain cross-modal alignment. Experimental results demonstrate that the fine-tuned model achieves more balanced and clinically meaningful performance across both image-to-text and text-to-image retrieval tasks compared to the pretrained Biomed

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliang2026multi

Related papers

• X-TRA: Improving Chest X-ray Tasks With Cross-modal Retrieval Augmentation (2023)8.09
• Medprobclip: Probabilistic Adaptation Of Vision-language Foundation Model For Reliable Radiograph-report Retrieval (2026)0.00
• Medclip: Contrastive Learning From Unpaired Medical Images And Text (2022)26.02
• Multi-level CLS Token Fusion For Contrastive Learning In Endoscopy Image Classification (2025)0.00
• Benchmarking Robustness Of Contrastive Learning Models For Medical Image-report Retrieval (2025)0.00
• Exploring The Capabilities Of LLM Encoders For Image-text Retrieval In Chest X-rays (2025)0.00
• A Comprehensive Empirical Study Of Vision-language Pre-trained Model For Supervised Cross-modal Retrieval (2022)0.00
• X-CLIP: End-to-end Multi-grained Contrastive Learning For Video-text Retrieval (2022)18.12