Linear Spaces Of Meanings: Compositional Structures In Vision-language Models

Abstract

We investigate compositional structures in data embeddings from pre-trained vision-language models (VLMs). Traditionally, compositionality has been associated with algebraic operations on embeddings of words from a pre-existing vocabulary. In contrast, we seek to approximate representations from an encoder as combinations of a smaller set of vectors in the embedding space. These vectors can be seen as "ideal words" for generating concepts directly within the embedding space of the model. We first present a framework for understanding compositional structures from a geometric perspective. We then explain what these compositional structures entail probabilistically in the case of VLM embeddings, providing intuitions for why they arise in practice. Finally, we empirically explore these structures in CLIP's embeddings and we evaluate their usefulness for solving different vision-language tasks such as classification, debiasing, and retrieval. Our results show that simple linear algebraic o

Stats

• citations17
• S2 citations—
• github stars0
• HF likes0
• heat score9.41
• arxiv keytrager2023linear

Related papers

• Prompting Large Vision-language Models For Compositional Reasoning (2024)0.00
• Adding Simple Structure At Inference Improves Vision-language Compositionality (2025)0.00
• Learning Visual Composition Through Improved Semantic Guidance (2024)0.00
• Compositional Image-text Matching And Retrieval By Grounding Entities (2025)0.60
• ARGENT: Adaptive Hierarchical Image-text Representations (2026)0.00
• Contrasting Intra-modal And Ranking Cross-modal Hard Negatives To Enhance Visio-linguistic Compositional Understanding (2023)12.11
• Lost In Embeddings: Information Loss In Vision-language Models (2025)0.00
• Semantic Compositions Enhance Vision-language Contrastive Learning (2024)0.00