Extending CLIP For Category-to-image Retrieval In E-commerce

Abstract

E-commerce provides rich multimodal data that is barely leveraged in practice. One aspect of this data is a category tree that is being used in search and recommendation. However, in practice, during a user's session there is often a mismatch between a textual and a visual representation of a given category. Motivated by the problem, we introduce the task of category-to-image retrieval in e-commerce and propose a model for the task, CLIP-ITA. The model leverages information from multiple modalities (textual, visual, and attribute modality) to create product representations. We explore how adding information from multiple modalities (textual, visual, and attribute modality) impacts the model's performance. In particular, we observe that CLIP-ITA significantly outperforms a comparable model that leverages only the visual modality and a comparable model that leverages the visual and attribute modality.

Stats

• citations13
• S2 citations—
• github stars0
• HF likes0
• heat score8.60
• arxiv keyhendriksen2021extending

Related papers

• VL-CLIP: Enhancing Multimodal Recommendations Via Visual Grounding And Llm-augmented CLIP Embeddings (2025)2.26
• Enhancing Image Retrieval : A Comprehensive Study On Photo Search Using The CLIP Mode (2024)0.00
• Turning Adversaries Into Allies: Reversing Typographic Attacks For Multimodal E-commerce Product Retrieval (2025)0.00
• Category-level Text-to-image Retrieval Improved: Bridging The Domain Gap With Diffusion Models And Vision Encoders (2025)1.20
• Captions Are Worth A Thousand Words: Enhancing Product Retrieval With Pretrained Image-to-text Models (2024)0.00
• ACE-BERT: Adversarial Cross-modal Enhanced BERT For E-commerce Retrieval (2021)0.00
• A Comprehensive Empirical Study Of Vision-language Pre-trained Model For Supervised Cross-modal Retrieval (2022)0.00
• Clip4clip: An Empirical Study Of CLIP For End To End Video Clip Retrieval (2021)6.02