Pebr: A Probabilistic Approach To Embedding Based Retrieval

Abstract

Embedding-based retrieval aims to learn a shared semantic representation space for both queries and items, enabling efficient and effective item retrieval through approximate nearest neighbor (ANN) algorithms. In current industrial practice, retrieval systems typically retrieve a fixed number of items for each query. However, this fixed-size retrieval often results in insufficient recall for head queries and low precision for tail queries. This limitation largely stems from the dominance of frequentist approaches in loss function design, which fail to address this challenge in industry. In this paper, we propose a novel \textbf\{p\}robabilistic \textbf\{E\}mbedding-\textbf\{B\}ased \textbf\{R\}etrieval (\textbf\{pEBR\}) framework. Our method models the item distribution conditioned on each query, enabling the use of a dynamic cosine similarity threshold derived from the cumulative distribution function (CDF) of the probabilistic model. Experimental results demonstrate that pEBR signifi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2024pebr

Related papers

• Divide And Conquer: Towards Better Embedding-based Retrieval For Recommender Systems From A Multi-task Perspective (2023)7.16
• Progressively Optimized Bi-granular Document Representation For Scalable Embedding Based Retrieval (2022)11.06
• Event-enhanced Retrieval In Real-time Search (2024)0.95
• CPS-MEBR: Click Feedback-aware Web Page Summarization For Multi-embedding-based Retrieval (2022)0.00
• Enhancing Relevance Of Embedding-based Retrieval At Walmart (2024)7.16
• ESANS: Effective And Semantic-aware Negative Sampling For Large-scale Retrieval Systems (2025)2.26
• PEFA: Parameter-free Adapters For Large-scale Embedding-based Retrieval Models (2023)7.73
• Unified Embedding Based Personalized Retrieval In Etsy Search (2023)2.26