LIDER: An Efficient High-dimensional Learned Index For Large-scale Dense Passage Retrieval

Abstract

Many recent approaches of passage retrieval are using dense embeddings generated from deep neural models, called "dense passage retrieval". The state-of-the-art end-to-end dense passage retrieval systems normally deploy a deep neural model followed by an approximate nearest neighbor (ANN) search module. The model generates embeddings of the corpus and queries, which are then indexed and searched by the high-performance ANN module. With the increasing data scale, the ANN module unavoidably becomes the bottleneck on efficiency. An alternative is the learned index, which achieves significantly high search efficiency by learning the data distribution and predicting the target data location. But most of the existing learned indexes are designed for low dimensional data, which are not suitable for dense passage retrieval with high-dimensional dense embeddings. In this paper, we propose LIDER, an efficient high-dimensional Learned Index for large-scale DEnse passage Retrieval. LIDER has a clu

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2022lider

Related papers

• EHI: End-to-end Learning Of Hierarchical Index For Efficient Dense Retrieval (2023)0.00
• Blending Learning To Rank And Dense Representations For Efficient And Effective Cascades (2025)0.00
• Densifying Sparse Representations For Passage Retrieval By Representational Slicing (2021)0.00
• Scaling Laws For Embedding Dimension In Information Retrieval (2026)0.00
• Gnn-encoder: Learning A Dual-encoder Architecture Via Graph Neural Networks For Dense Passage Retrieval (2022)4.52
• Large Reasoning Embedding Models: Towards Next-generation Dense Retrieval Paradigm (2025)0.00
• I^3 Retriever: Incorporating Implicit Interaction In Pre-trained Language Models For Passage Retrieval (2023)7.16
• Hierarchical Corpus Encoder: Fusing Generative Retrieval And Dense Indices (2025)0.00