X-TRA: Improving Chest X-ray Tasks With Cross-modal Retrieval Augmentation

Abstract

An important component of human analysis of medical images and their context is the ability to relate newly seen things to related instances in our memory. In this paper we mimic this ability by using multi-modal retrieval augmentation and apply it to several tasks in chest X-ray analysis. By retrieving similar images and/or radiology reports we expand and regularize the case at hand with additional knowledge, while maintaining factual knowledge consistency. The method consists of two components. First, vision and language modalities are aligned using a pre-trained CLIP model. To enforce that the retrieval focus will be on detailed disease-related content instead of global visual appearance it is fine-tuned using disease class information. Subsequently, we construct a non-parametric retrieval index, which reaches state-of-the-art retrieval levels. We use this index in our downstream tasks to augment image representations through multi-head attention for disease classification and repor

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keyvansonsbeek2023x

Related papers

• Multi-task Cross-modal Learning For Chest X-ray Image Retrieval (2026)0.00
• Multimodal Image-text Matching Improves Retrieval-based Chest X-ray Report Generation (2023)3.33
• Cross-modality Sub-image Retrieval Using Contrastive Multimodal Image Representations (2022)6.32
• Cross-modal Retrieval Augmentation For Multi-modal Classification (2021)9.23
• Radir: A Scalable Framework For Multi-grained Medical Image Retrieval Via Radiology Report Mining (2025)0.00
• Unsupervised Multimodal Representation Learning Across Medical Images And Reports (2018)0.00
• Prototype-enhanced Confidence Modeling For Cross-modal Medical Image-report Retrieval (2025)0.00
• More: Multi-modal Contrastive Pre-training With Transformers On X-rays, Ecgs, And Diagnostic Report (2024)0.00