Learning Visual Composition Through Improved Semantic Guidance

Abstract

Visual imagery does not consist of solitary objects, but instead reflects the composition of a multitude of fluid concepts. While there have been great advances in visual representation learning, such advances have focused on building better representations for a small number of discrete objects bereft of an understanding of how these objects are interacting. One can observe this limitation in representations learned through captions or contrastive learning -- where the learned model treats an image essentially as a bag of words. Several works have attempted to address this limitation through the development of bespoke learned architectures to directly address the shortcomings in compositional learning. In this work, we focus on simple, and scalable approaches. In particular, we demonstrate that by substantially improving weakly labeled data, i.e. captions, we can vastly improve the performance of standard contrastive learning approaches. Previous CLIP models achieved near chance rate

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keystone2024learning

Related papers

• Semantic Compositions Enhance Vision-language Contrastive Learning (2024)0.00
• Advancing Compositional Awareness In CLIP With Efficient Fine-tuning (2025)0.00
• Contrasting Intra-modal And Ranking Cross-modal Hard Negatives To Enhance Visio-linguistic Compositional Understanding (2023)12.11
• Prompting Large Vision-language Models For Compositional Reasoning (2024)0.00
• Adding Simple Structure At Inference Improves Vision-language Compositionality (2025)0.00
• Composed Image Retrieval Using Contrastive Learning And Task-oriented Clip-based Features (2023)16.84
• Tripletclip: Improving Compositional Reasoning Of CLIP Via Synthetic Vision-language Negatives (2024)4.52
• COLA: A Benchmark For Compositional Text-to-image Retrieval (2023)0.00