Anatomy-aware Conditional Image-text Retrieval

Abstract

Image-Text Retrieval (ITR) finds broad applications in healthcare, aiding clinicians and radiologists by automatically retrieving relevant patient cases in the database given the query image and/or report, for more efficient clinical diagnosis and treatment, especially for rare diseases. However conventional ITR systems typically only rely on global image or text representations for measuring patient image/report similarities, which overlook local distinctiveness across patient cases. This often results in suboptimal retrieval performance. In this paper, we propose an Anatomical Location-Conditioned Image-Text Retrieval (ALC-ITR) framework, which, given a query image and the associated suspicious anatomical region(s), aims to retrieve similar patient cases exhibiting the same disease or symptoms in the same anatomical region. To perform location-conditioned multimodal retrieval, we learn a medical Relevance-Region-Aligned Vision Language (RRA-VL) model with semantic global-level and re

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzheng2025anatomy

Related papers

• Image-text Retrieval: A Survey On Recent Research And Development (2022)13.93
• HGAN: Hierarchical Graph Alignment Network For Image-text Retrieval (2022)11.93
• Benchmark Granularity And Model Robustness For Image-text Retrieval (2024)0.00
• Region-based Contrastive Pretraining For Medical Image Retrieval With Anatomic Query (2023)0.00
• X-TRA: Improving Chest X-ray Tasks With Cross-modal Retrieval Augmentation (2023)8.09
• Integrating Listwise Ranking Into Pairwise-based Image-text Retrieval (2023)9.16
• Language Guided Local Infiltration For Interactive Image Retrieval (2023)5.24
• Active Learning For Finely-categorized Image-text Retrieval By Selecting Hard Negative Unpaired Samples (2024)2.26