Mobileclip: Fast Image-text Models Through Multi-modal Reinforced Training

Abstract

Contrastive pretraining of image-text foundation models, such as CLIP, demonstrated excellent zero-shot performance and improved robustness on a wide range of downstream tasks. However, these models utilize large transformer-based encoders with significant memory and latency overhead which pose challenges for deployment on mobile devices. In this work, we introduce MobileCLIP -- a new family of efficient image-text models optimized for runtime performance along with a novel and efficient training approach, namely multi-modal reinforced training. The proposed training approach leverages knowledge transfer from an image captioning model and an ensemble of strong CLIP encoders to improve the accuracy of efficient models. Our approach avoids train-time compute overhead by storing the additional knowledge in a reinforced dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for zero-shot classification and retrieval tasks on several datasets. Our MobileCLIP-S2 variant is

Stats

• citations36
• S2 citations—
• github stars1518
• HF likes0
• heat score18.12
• arxiv keyvasu2023mobileclip

Related papers

• Mobileviclip: An Efficient Video-text Model For Mobile Devices (2025)2.76
• Jina CLIP: Your CLIP Model Is Also Your Text Retriever (2024)0.00
• Efficientclip: Efficient Cross-modal Pre-training By Ensemble Confident Learning And Language Modeling (2021)0.00
• RECLIP: Resource-efficient CLIP By Training With Small Images (2023)0.00
• Clip-lite: Information Efficient Visual Representation Learning With Language Supervision (2021)2.35
• Contextclip: Contextual Alignment Of Image-text Pairs On CLIP Visual Representations (2022)5.84
• Clip2video: Mastering Video-text Retrieval Via Image CLIP (2021)0.00
• Distill CLIP (DCLIP): Enhancing Image-text Retrieval Via Cross-modal Transformer Distillation (2025)0.00