Robust Cross-modal Representation Learning With Progressive Self-distillation

Abstract

The learning objective of vision-language approach of CLIP does not effectively account for the noisy many-to-many correspondences found in web-harvested image captioning datasets, which contributes to its compute and data inefficiency. To address this challenge, we introduce a novel training framework based on cross-modal contrastive learning that uses progressive self-distillation and soft image-text alignments to more efficiently learn robust representations from noisy data. Our model distills its own knowledge to dynamically generate soft-alignment targets for a subset of images and captions in every minibatch, which are then used to update its parameters. Extensive evaluation across 14 benchmark datasets shows that our method consistently outperforms its CLIP counterpart in multiple settings, including: (a) zero-shot classification, (b) linear probe transfer, and (c) image-text retrieval, without incurring added computational cost. Analysis using an ImageNet-based robustness test-

Stats

• citations43
• S2 citations—
• github stars0
• HF likes0
• heat score12.33
• arxiv keyandonian2022robust

Related papers

• SILC: Improving Vision Language Pretraining With Self-distillation (2023)10.21
• Clip-lite: Information Efficient Visual Representation Learning With Language Supervision (2021)2.35
• Advancing Myopia To Holism: Fully Contrastive Language-image Pre-training (2024)0.00
• CIBR: Cross-modal Information Bottleneck Regularization For Robust CLIP Generalization (2025)4.52
• Distill CLIP (DCLIP): Enhancing Image-text Retrieval Via Cross-modal Transformer Distillation (2025)0.00
• CLIPS: An Enhanced CLIP Framework For Learning With Synthetic Captions (2024)0.00
• Linear Alignment Of Vision-language Models For Image Captioning (2023)0.00
• Contextclip: Contextual Alignment Of Image-text Pairs On CLIP Visual Representations (2022)5.84