Vision-language Modelling For Radiological Imaging And Reports In The Low Data Regime

Abstract

This paper explores training medical vision-language models (VLMs) -- where the visual and language inputs are embedded into a common space -- with a particular focus on scenarios where training data is limited, as is often the case in clinical datasets. We explore several candidate methods to improve low-data performance, including: (i) adapting generic pre-trained models to novel image and text domains (i.e. medical imaging and reports) via unimodal self-supervision; (ii) using local (e.g. GLoRIA) & global (e.g. InfoNCE) contrastive loss functions as well as a combination of the two; (iii) extra supervision during VLM training, via: (a) image- and text-only self-supervision, and (b) creating additional positive image-text pairs for training through augmentation and nearest-neighbour search. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports. Combined, they s

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywindsor2023vision

Related papers

• Lvlm-aware Multimodal Retrieval For Rag-based Medical Diagnosis With General-purpose Models (2025)0.00
• Med3dvlm: An Efficient Vision-language Model For 3D Medical Image Analysis (2025)12.60
• Learning To Read Where To Look: Disease-aware Vision-language Pretraining For 3D CT (2026)0.00
• Exploring The Capabilities Of LLM Encoders For Image-text Retrieval In Chest X-rays (2025)0.00
• Villa: Fine-grained Vision-language Representation Learning From Real-world Data (2023)8.82
• Benchmarking Vision-language Contrastive Methods For Medical Representation Learning (2024)0.00
• Addressing Bias In Vlms For Glaucoma Detection Without Protected Attribute Supervision (2025)0.00
• Efficient Medical Vision-language Alignment Through Adapting Masked Vision Models (2025)5.74