Cross-modality Sub-image Retrieval Using Contrastive Multimodal Image Representations

Abstract

In tissue characterization and cancer diagnostics, multimodal imaging has emerged as a powerful technique. Thanks to computational advances, large datasets can be exploited to discover patterns in pathologies and improve diagnosis. However, this requires efficient and scalable image retrieval methods. Cross-modality image retrieval is particularly challenging, since images of similar (or even the same) content captured by different modalities might share few common structures. We propose a new application-independent content-based image retrieval (CBIR) system for reverse (sub-)image search across modalities, which combines deep learning to generate representations (embedding the different modalities in a common space) with classical feature extraction and bag-of-words models for efficient and reliable retrieval. We illustrate its advantages through a replacement study, exploring a number of feature extractors and learned representations, as well as through comparison to recent (cross-

Stats

• citations3
• S2 citations—
• github stars7
• HF likes0
• heat score6.32
• arxiv keybreznik2022cross

Related papers

• X-TRA: Improving Chest X-ray Tasks With Cross-modal Retrieval Augmentation (2023)8.09
• Content-based 3D Image Retrieval And A Colbert-inspired Re-ranking For Tumor Flagging And Staging (2025)0.00
• CMIR-NET : A Deep Learning Based Model For Cross-modal Retrieval In Remote Sensing (2019)13.34
• Revisiting Cross Modal Retrieval (2018)0.00
• MRIS: A Multi-modal Retrieval Approach For Image Synthesis On Diverse Modalities (2023)3.58
• A Novel Self-supervised Cross-modal Image Retrieval Method In Remote Sensing (2022)8.35
• Leveraging Foundation Models For Content-based Image Retrieval In Radiology (2024)11.11
• Multi-task Cross-modal Learning For Chest X-ray Image Retrieval (2026)0.00