SILC: Improving Vision Language Pretraining With Self-distillation

Abstract

Image-Text pretraining on web-scale image caption datasets has become the default recipe for open vocabulary classification and retrieval models thanks to the success of CLIP and its variants. Several works have also used CLIP features for dense prediction tasks and have shown the emergence of open-set abilities. However, the contrastive objective used by these models only focuses on image-text alignment and does not incentivise image feature learning for dense prediction tasks. In this work, we introduce SILC, a novel framework for vision language pretraining. SILC improves image-text contrastive learning with the simple addition of local-to-global correspondence learning by self-distillation. We show that distilling local image features from an exponential moving average (EMA) teacher model significantly improves model performance on dense predictions tasks like detection and segmentation, while also providing improvements on image-level tasks such as classification and retrieval. SI

Stats

• citations22
• S2 citations—
• github stars0
• HF likes0
• heat score10.21
• arxiv keynaeem2023silc

Related papers

• Robust Cross-modal Representation Learning With Progressive Self-distillation (2022)12.33
• Modeling Caption Diversity In Contrastive Vision-language Pretraining (2024)0.00
• Distill CLIP (DCLIP): Enhancing Image-text Retrieval Via Cross-modal Transformer Distillation (2025)0.00
• CLIP-KD: An Empirical Study Of CLIP Model Distillation (2023)17.57
• ELIP: Enhanced Visual-language Foundation Models For Image Retrieval (2025)2.26
• Goldiclip: The Goldilocks Approach For Balancing Explicit Supervision For Language-image Pretraining (2026)0.00
• Clip-lite: Information Efficient Visual Representation Learning With Language Supervision (2021)2.35
• Dynamic Contrastive Distillation For Image-text Retrieval (2022)11.76