Bringing Multimodality To Amazon Visual Search System

Abstract

Image to image matching has been well studied in the computer vision community. Previous studies mainly focus on training a deep metric learning model matching visual patterns between the query image and gallery images. In this study, we show that pure image-to-image matching suffers from false positives caused by matching to local visual patterns. To alleviate this issue, we propose to leverage recent advances in vision-language pretraining research. Specifically, we introduce additional image-text alignment losses into deep metric learning, which serve as constraints to the image-to-image matching loss. With additional alignments between the text (e.g., product title) and image pairs, the model can learn concepts from both modalities explicitly, which avoids matching low-level visual features. We progressively develop two variants, a 3-tower and a 4-tower model, where the latter takes one more short text query input. Through extensive experiments, we show that this change leads to a

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyzhu2024bringing

Related papers

• Multimodal Representation Alignment For Cross-modal Information Retrieval (2025)0.00
• MAKE: Vision-language Pre-training Based Product Retrieval In Taobao Search (2023)7.81
• Embedding Arithmetic Of Multimodal Queries For Image Retrieval (2021)9.03
• Turning Adversaries Into Allies: Reversing Typographic Attacks For Multimodal E-commerce Product Retrieval (2025)0.00
• Deep Multimodal Image-text Embeddings For Automatic Cross-media Retrieval (2020)0.00
• MASS: Overcoming Language Bias In Image-text Matching (2025)0.00
• Indexing Multimodal Language Models For Large-scale Image Retrieval (2026)0.00
• Mllms-augmented Visual-language Representation Learning (2023)0.00