Multimodal Contrastive Training For Visual Representation Learning

Abstract

We develop an approach to learning visual representations that embraces multimodal data, driven by a combination of intra- and inter-modal similarity preservation objectives. Unlike existing visual pre-training methods, which solve a proxy prediction task in a single domain, our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations. By including multimodal training in a unified framework with different types of contrastive losses, our method can learn more powerful and generic visual features. We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation. For example, the visual representations pre-trained on COCO by our method achieve state-of-the-art top-1 validation accuracy of \(55.3%\) on ImageNet classification, under th

Stats

• citations149
• S2 citations—
• github stars0
• HF likes0
• heat score16.32
• arxiv keyyuan2021multimodal

Related papers

• Revisiting Contrastive Methods For Unsupervised Learning Of Visual Representations (2021)3.91
• Compressing Then Matching: An Efficient Pre-training Paradigm For Multimodal Embedding (2025)0.00
• Contrastive Cross-modal Knowledge Sharing Pre-training For Vision-language Representation Learning And Retrieval (2022)0.00
• Explaining And Mitigating The Modality Gap In Contrastive Multimodal Learning (2024)0.00
• Contrastive Learning Of Visual-semantic Embeddings (2021)0.00
• Multimodal Representation Learning Conditioned On Semantic Relations (2025)0.00
• CAVL: Learning Contrastive And Adaptive Representations Of Vision And Language (2023)0.00
• Using Multiple Instance Learning To Build Multimodal Representations (2022)4.52