Curriculum Learning For Data-efficient Vision-language Alignment

Abstract

Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation models using a much smaller amount of paired data, augmented with a curriculum learning algorithm to learn fine-grained vision-language alignments. TOnICS (Training with Ontology-Informed Contrastive Sampling) initially samples minibatches whose image-text pairs contain a wide variety of objects to learn object-level alignment, and progressively samples minibatches where all image-text pairs contain the same object to learn finer-grained contextual alignment. Aligning pre-trained BERT and VinVL models to each other using TOnICS outperforms CLIP on downstream zero-shot image retrieval while using less than 1% as much training data.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keysrinivasan2022curriculum

Related papers

• COTS: Collaborative Two-stream Vision-language Pre-training Model For Cross-modal Retrieval (2022)13.60
• Modest-align: Data-efficient Alignment For Vision-language Models (2025)0.00
• Linear Alignment Of Vision-language Models For Image Captioning (2023)0.00
• Advancing Myopia To Holism: Fully Contrastive Language-image Pre-training (2024)0.00
• Robust Cross-modal Representation Learning With Progressive Self-distillation (2022)12.33
• CAVL: Learning Contrastive And Adaptive Representations Of Vision And Language (2023)0.00
• Uclip: Parameter-efficient Multilingual Extension Of Vision-language Models With Unpaired Data (2025)0.00
• GOAL: Global-local Object Alignment Learning (2025)2.26