Learning Visual-semantic Embeddings For Reporting Abnormal Findings On Chest X-rays

Abstract

Automatic medical image report generation has drawn growing attention due to its potential to alleviate radiologists' workload. Existing work on report generation often trains encoder-decoder networks to generate complete reports. However, such models are affected by data bias (e.g.~label imbalance) and face common issues inherent in text generation models (e.g.~repetition). In this work, we focus on reporting abnormal findings on radiology images; instead of training on complete radiology reports, we propose a method to identify abnormal findings from the reports in addition to grouping them with unsupervised clustering and minimal rules. We formulate the task as cross-modal retrieval and propose Conditional Visual-Semantic Embeddings to align images and fine-grained abnormal findings in a joint embedding space. We demonstrate that our method is able to retrieve abnormal findings and outperforms existing generation models on both clinical correctness and text generation metrics.

Stats

• citations19
• S2 citations—
• github stars0
• HF likes0
• heat score9.76
• arxiv keyni2020learning

Related papers

• Multimodal Image-text Matching Improves Retrieval-based Chest X-ray Report Generation (2023)3.33
• Unsupervised Multimodal Representation Learning Across Medical Images And Reports (2018)0.00
• Exploring The Capabilities Of LLM Encoders For Image-text Retrieval In Chest X-rays (2025)0.00
• Grounded Multimodal Retrieval-augmented Drafting Of Radiology Impressions Using Case-based Similarity Search (2026)0.00
• Vision-language Modelling For Radiological Imaging And Reports In The Low Data Regime (2023)0.00
• Beyond The Embedding Bottleneck: Adaptive Retrieval-augmented 3D CT Report Generation (2026)0.00
• Ontology-based Concept Distillation For Radiology Report Retrieval And Labeling (2025)1.20
• X-TRA: Improving Chest X-ray Tasks With Cross-modal Retrieval Augmentation (2023)8.09