ASIF: Coupled Data Turns Unimodal Models To Multimodal Without Training

Abstract

CLIP proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of image-text pairs. Furthermore, our model has unique properties. Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique image-text pair in the multimodal dataset. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keynorelli2022asif

Related papers

• Cross The Gap: Exposing The Intra-modal Misalignment In CLIP Via Modality Inversion (2025)3.64
• Hyperdimensional Cross-modal Alignment Of Frozen Language And Image Models For Efficient Image Captioning (2026)0.00
• CSA: Data-efficient Mapping Of Unimodal Features To Multimodal Features (2024)0.00
• Breaking The Modality Barrier: Universal Embedding Learning With Multimodal Llms (2025)4.52
• Modest-align: Data-efficient Alignment For Vision-language Models (2025)0.00
• Generating Images With Multimodal Language Models (2023)6.77
• Isoclip: Decomposing CLIP Projectors For Efficient Intra-modal Alignment (2026)3.06
• Image Search Using Multilingual Texts: A Cross-modal Learning Approach Between Image And Text (2019)0.00