Divide And Conquer: Towards Better Embedding-based Retrieval For Recommender Systems From A Multi-task Perspective

Abstract

Embedding-based retrieval (EBR) methods are widely used in modern recommender systems thanks to its simplicity and effectiveness. However, along the journey of deploying and iterating on EBR in production, we still identify some fundamental issues in existing methods. First, when dealing with large corpus of candidate items, EBR models often have difficulties in balancing the performance on distinguishing highly relevant items (positives) from both irrelevant ones (easy negatives) and from somewhat related yet not competitive ones (hard negatives). Also, we have little control in the diversity and fairness of the retrieval results because of the ``greedy'' nature of nearest vector search. These issues compromise the performance of EBR methods in large-scale industrial scenarios. This paper introduces a simple and proven-in-production solution to overcome these issues. The proposed solution takes a divide-and-conquer approach: the whole set of candidate items are divided into multiple c

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keyzhang2023divide

Related papers

• Robust Training Objectives Improve Embedding-based Retrieval In Industrial Recommendation Systems (2024)0.00
• Hierarchical Structured Neural Network: Efficient Retrieval Scaling For Large Scale Recommendation (2024)0.00
• Enhancing Relevance Of Embedding-based Retrieval At Walmart (2024)7.16
• Pebr: A Probabilistic Approach To Embedding Based Retrieval (2024)0.00
• Embedding In Recommender Systems: A Survey (2023)0.00
• CSMF: Cascaded Selective Mask Fine-tuning For Multi-objective Embedding-based Retrieval (2025)0.00
• Embedding-based Recommender System For Job To Candidate Matching On Scale (2021)0.00
• CPS-MEBR: Click Feedback-aware Web Page Summarization For Multi-embedding-based Retrieval (2022)0.00