Progressively Optimized Bi-granular Document Representation For Scalable Embedding Based Retrieval

Abstract

Ad-hoc search calls for the selection of appropriate answers from a massive-scale corpus. Nowadays, the embedding-based retrieval (EBR) becomes a promising solution, where deep learning based document representation and ANN search techniques are allied to handle this task. However, a major challenge is that the ANN index can be too large to fit into memory, given the considerable size of answer corpus. In this work, we tackle this problem with Bi-Granular Document Representation, where the lightweight sparse embeddings are indexed and standby in memory for coarse-grained candidate search, and the heavyweight dense embeddings are hosted in disk for fine-grained post verification. For the best of retrieval accuracy, a Progressive Optimization framework is designed. The sparse embeddings are learned ahead for high-quality search of candidates. Conditioned on the candidate distribution induced by the sparse embeddings, the dense embeddings are continuously learned to optimize the discrimin

Stats

• citations13
• S2 citations—
• github stars16
• HF likes0
• heat score11.06
• arxiv keyxiao2022progressively

Related papers

• Pebr: A Probabilistic Approach To Embedding Based Retrieval (2024)0.00
• Pre-training Tasks For Embedding-based Large-scale Retrieval (2020)0.00
• EHI: End-to-end Learning Of Hierarchical Index For Efficient Dense Retrieval (2023)0.00
• Hierarchical Structured Neural Network: Efficient Retrieval Scaling For Large Scale Recommendation (2024)0.00
• Improving Document Representations By Generating Pseudo Query Embeddings For Dense Retrieval (2021)9.41
• Efficient Inverted Indexes For Approximate Retrieval Over Learned Sparse Representations (2024)11.67
• Event-enhanced Retrieval In Real-time Search (2024)0.95
• Efficient And Effective Retrieval Of Dense-sparse Hybrid Vectors Using Graph-based Approximate Nearest Neighbor Search (2024)0.00