Improving Visual Recommendation On E-commerce Platforms Using Vision-language Models

Abstract

On large-scale e-commerce platforms with tens of millions of active monthly users, recommending visually similar products is essential for enabling users to efficiently discover items that align with their preferences. This study presents the application of a vision-language model (VLM) -- which has demonstrated strong performance in image recognition and image-text retrieval tasks -- to product recommendations on Mercari, a major consumer-to-consumer marketplace used by more than 20 million monthly users in Japan. Specifically, we fine-tuned SigLIP, a VLM employing a sigmoid-based contrastive loss, using one million product image-title pairs from Mercari collected over a three-month period, and developed an image encoder for generating item embeddings used in the recommendation system. Our evaluation comprised an offline analysis of historical interaction logs and an online A/B test in a production environment. In offline analysis, the model achieved a 9.1% improvement in nDCG@5 compa

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyada2025improving

Related papers

• Zero-shot Retrieval For Scalable Visual Search In A Two-sided Marketplace (2025)1.57
• VL-CLIP: Enhancing Multimodal Recommendations Via Visual Grounding And Llm-augmented CLIP Embeddings (2025)2.26
• V\(^2\)L: Leveraging Vision And Vision-language Models Into Large-scale Product Retrieval (2022)0.00
• Vlm4rec: Multimodal Semantic Representation For Recommendation With Large Vision-language Models (2026)1.82
• Deep Learning Based Large Scale Visual Recommendation And Search For E-commerce (2017)0.00
• An Analysis Of Vision-language Models For Fabric Retrieval (2025)0.00
• Factorized Transport Alignment For Multimodal And Multiview E-commerce Representation Learning (2025)0.00
• Delving Into E-commerce Product Retrieval With Vision-language Pre-training (2023)6.77