Exploring The Capabilities Of LLM Encoders For Image-text Retrieval In Chest X-rays

Abstract

Vision-language pretraining has advanced image-text alignment, yet progress in radiology remains constrained by the heterogeneity of clinical reports, including abbreviations, impression-only notes, and stylistic variability. Unlike general-domain settings where more data often leads to better performance, naively scaling to large collections of noisy reports can plateau or even degrade model learning. We ask whether large language model (LLM) encoders can provide robust clinical representations that transfer across diverse styles and better guide image-text alignment. We introduce LLM2VEC4CXR, a domain-adapted LLM encoder for chest X-ray reports, and LLM2CLIP4CXR, a dual-tower framework that couples this encoder with a vision backbone. LLM2VEC4CXR improves clinical text understanding over BERT-based baselines, handles abbreviations and style variation, and achieves strong clinical alignment on report-level metrics. LLM2CLIP4CXR leverages these embeddings to boost retrieval accuracy an

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyko2025exploring

Related papers

• Multi-task Cross-modal Learning For Chest X-ray Image Retrieval (2026)0.00
• Vision-language Modelling For Radiological Imaging And Reports In The Low Data Regime (2023)0.00
• Medprobclip: Probabilistic Adaptation Of Vision-language Foundation Model For Reliable Radiograph-report Retrieval (2026)0.00
• Multi-level CLS Token Fusion For Contrastive Learning In Endoscopy Image Classification (2025)0.00
• Learning To Read Where To Look: Disease-aware Vision-language Pretraining For 3D CT (2026)0.00
• Learning Visual-semantic Embeddings For Reporting Abnormal Findings On Chest X-rays (2020)9.76
• Med3dvlm: An Efficient Vision-language Model For 3D Medical Image Analysis (2025)12.60
• LLM2CLIP: Powerful Language Model Unlocks Richer Cross-modality Representation (2024)2.26