Efficient Medical Vision-language Alignment Through Adapting Masked Vision Models

Abstract

Medical vision-language alignment through cross-modal contrastive learning shows promising performance in image-text matching tasks, such as retrieval and zero-shot classification. However, conventional cross-modal contrastive learning (CLIP-based) methods suffer from suboptimal visual representation capabilities, which also limits their effectiveness in vision-language alignment. In contrast, although the models pretrained via multimodal masked modeling struggle with direct cross-modal matching, they excel in visual representation. To address this contradiction, we propose ALTA (ALign Through Adapting), an efficient medical vision-language alignment method that utilizes only about 8% of the trainable parameters and less than 1/5 of the computational consumption required for masked record modeling. ALTA achieves superior performance in vision-language matching tasks like retrieval and zero-shot classification by adapting the pretrained vision model from masked record modeling. Addition

Stats

• citations2
• S2 citations—
• github stars11
• HF likes0
• heat score5.74
• arxiv keylian2025efficient

Related papers

• Masked Contrastive Reconstruction For Cross-modal Medical Image-report Retrieval (2023)0.00
• Medclip: Contrastive Learning From Unpaired Medical Images And Text (2022)26.02
• Uclip: Parameter-efficient Multilingual Extension Of Vision-language Models With Unpaired Data (2025)0.00
• Modest-align: Data-efficient Alignment For Vision-language Models (2025)0.00
• Benchmarking Vision-language Contrastive Methods For Medical Representation Learning (2024)0.00
• Multi-task Cross-modal Learning For Chest X-ray Image Retrieval (2026)0.00
• Advancing Myopia To Holism: Fully Contrastive Language-image Pre-training (2024)0.00
• VLMAE: Vision-language Masked Autoencoder (2022)0.00