From Pixels To Purchase: Building And Evaluating A Taxonomy-decoupled Visual Search Engine For Home Goods E-commerce

Abstract

Visual search is critical for e-commerce, especially in style-driven domains where user intent is subjective and open-ended. Existing industrial systems typically couple object detection with taxonomy-based classification and rely on catalog data for evaluation, which is prone to noise that limits robustness and scalability. We propose a taxonomy-decoupled architecture that uses classification-free region proposals and unified embeddings for similarity retrieval, enabling a more flexible and generalizable visual search. To overcome the evaluation bottleneck, we propose an LLM-as-a-Judge framework that assesses nuanced visual similarity and category relevance for query-result pairs in a zero-shot manner, removing dependence on human annotations or noise-prone catalog data. Deployed at scale on a global home goods platform, our system improves retrieval quality and yields a measurable uplift in customer engagement, while our offline evaluation metrics strongly correlate with real-world o

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylyu2026from

Related papers

• Zero-shot Retrieval For Scalable Visual Search In A Two-sided Marketplace (2025)1.57
• Towards Practical Visual Search Engine Within Elasticsearch (2018)0.00
• Visually Similar Products Retrieval For Shopsy (2022)2.26
• Deep Learning Based Large Scale Visual Recommendation And Search For E-commerce (2017)0.00
• Visual Product Graph: Bridging Visual Products And Composite Images For End-to-end Style Recommendations (2025)0.00
• Onevision: An End-to-end Generative Framework For Multi-view E-commerce Vision Search (2025)0.00
• Taxonomy-based Negative Sampling In Personalized Semantic Search For E-commerce (2025)0.00
• The Design And Implementation Of A Real Time Visual Search System On JD E-commerce Platform (2019)12.25