Ultra-high Dimensional Sparse Representations With Binarization For Efficient Text Retrieval

Abstract

The semantic matching capabilities of neural information retrieval can ameliorate synonymy and polysemy problems of symbolic approaches. However, neural models' dense representations are more suitable for re-ranking, due to their inefficiency. Sparse representations, either in symbolic or latent form, are more efficient with an inverted index. Taking the merits of the sparse and dense representations, we propose an ultra-high dimensional (UHD) representation scheme equipped with directly controllable sparsity. UHD's large capacity and minimal noise and interference among the dimensions allow for binarized representations, which are highly efficient for storage and search. Also proposed is a bucketing method, where the embeddings from multiple layers of BERT are selected/merged to represent diverse linguistic aspects. We test our models with MS MARCO and TREC CAR, showing that our models outperforms other sparse models

Stats

• citations13
• S2 citations—
• github stars0
• HF likes0
• heat score8.60
• arxiv keyjang2021ultra

Related papers

• Efficient Inverted Indexes For Approximate Retrieval Over Learned Sparse Representations (2024)11.67
• A Dense Representation Framework For Lexical And Semantic Matching (2022)11.13
• Pseudo-relevance Feedback For Multiple Representation Dense Retrieval (2021)12.93
• SEINE: Segment-based Indexing For Neural Information Retrieval (2023)0.00
• Scaling Laws For Embedding Dimension In Information Retrieval (2026)0.00
• End-to-end Retrieval With Learned Dense And Sparse Representations Using Lucene (2023)0.00
• STAIR: Learning Sparse Text And Image Representation In Grounded Tokens (2023)8.82
• Deep Binaries: Encoding Semantic-rich Cues For Efficient Textual-visual Cross Retrieval (2017)12.10