Jina CLIP: Your CLIP Model Is Also Your Text Retriever

Abstract

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks. However, CLIP models generally underperform in text-only tasks compared to specialized text models. This creates inefficiencies for information retrieval systems that keep separate embeddings and models for text-only and multimodal tasks. We propose a novel, multi-task contrastive training method to address this issue, which we use to train the jina-clip-v1 model to achieve the state-of-the-art performance on both text-image and text-text retrieval tasks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykoukounas2024jina

Related papers

• Mobileclip: Fast Image-text Models Through Multi-modal Reinforced Training (2023)18.12
• Optimizing CLIP Models For Image Retrieval With Maintained Joint-embedding Alignment (2024)6.34
• Clip2video: Mastering Video-text Retrieval Via Image CLIP (2021)0.00
• Clip4clip: An Empirical Study Of CLIP For End To End Video Clip Retrieval (2021)6.02
• Contextclip: Contextual Alignment Of Image-text Pairs On CLIP Visual Representations (2022)5.84
• Finetuning CLIP To Reason About Pairwise Differences (2024)0.00
• Distill CLIP (DCLIP): Enhancing Image-text Retrieval Via Cross-modal Transformer Distillation (2025)0.00
• Enhancing Image Retrieval : A Comprehensive Study On Photo Search Using The CLIP Mode (2024)0.00