MRIS: A Multi-modal Retrieval Approach For Image Synthesis On Diverse Modalities

Abstract

Multiple imaging modalities are often used for disease diagnosis, prediction, or population-based analyses. However, not all modalities might be available due to cost, different study designs, or changes in imaging technology. If the differences between the types of imaging are small, data harmonization approaches can be used; for larger changes, direct image synthesis approaches have been explored. In this paper, we develop an approach based on multi-modal metric learning to synthesize images of diverse modalities. We use metric learning via multi-modal image retrieval, resulting in embeddings that can relate images of different modalities. Given a large image database, the learned image embeddings allow us to use k-nearest neighbor (k-NN) regression for image synthesis. Our driving medical problem is knee osteoarthritis (KOA), but our developed method is general after proper image alignment. We test our approach by synthesizing cartilage thickness maps obtained from 3D magnetic reson

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keychen2023mris

Related papers

• Cross-modality Sub-image Retrieval Using Contrastive Multimodal Image Representations (2022)6.32
• Unsupervised Multimodal Representation Learning Across Medical Images And Reports (2018)0.00
• Multimorbidity Content-based Medical Image Retrieval Using Proxies (2022)0.00
• Universal Model For Multi-domain Medical Image Retrieval (2020)0.00
• A Novel Self-supervised Cross-modal Image Retrieval Method In Remote Sensing (2022)8.35
• M3ret: Unleashing Zero-shot Multimodal Medical Image Retrieval Via Self-supervision (2025)0.00
• Cross-modal Retrieval And Synthesis (X-MRS): Closing The Modality Gap In Shared Representation Learning (2020)0.00
• X-TRA: Improving Chest X-ray Tasks With Cross-modal Retrieval Augmentation (2023)8.09