STAIR: Learning Sparse Text And Image Representation In Grounded Tokens

Abstract

Image and text retrieval is one of the foundational tasks in the vision and language domain with multiple real-world applications. State-of-the-art approaches, e.g. CLIP, ALIGN, represent images and texts as dense embeddings and calculate the similarity in the dense embedding space as the matching score. On the other hand, sparse semantic features like bag-of-words models are more interpretable, but believed to suffer from inferior accuracy than dense representations. In this work, we show that it is possible to build a sparse semantic representation that is as powerful as, or even better than, dense presentations. We extend the CLIP model and build a sparse text and image representation (STAIR), where the image and text are mapped to a sparse token space. Each token in the space is a (sub-)word in the vocabulary, which is not only interpretable but also easy to integrate with existing information retrieval systems. STAIR model significantly outperforms a CLIP model with +\(4.9%\) and

Stats

• citations14
• S2 citations—
• github stars0
• HF likes0
• heat score8.82
• arxiv keychen2023stair

Related papers

• Sparse And Dense Retrievers Learn Better Together: Joint Sparse-dense Optimization For Text-image Retrieval (2025)0.00
• Semantic Token Reweighting For Interpretable And Controllable Text Embeddings In CLIP (2024)0.00
• Ultra-high Dimensional Sparse Representations With Binarization For Efficient Text Retrieval (2021)8.60
• FLAIR: VLM With Fine-grained Language-informed Image Representations (2024)10.14
• Improving Fine-grained Understanding In Image-text Pre-training (2024)0.00
• Lexlip: Lexicon-bottlenecked Language-image Pre-training For Large-scale Image-text Retrieval (2023)10.85
• Text-video Retrieval With Global-local Semantic Consistent Learning (2024)8.75
• Stacked Cross Attention For Image-text Matching (2018)27.77