Rethinking The Role Of Token Retrieval In Multi-vector Retrieval

Abstract

Multi-vector retrieval models such as ColBERT [Khattab and Zaharia, 2020] allow token-level interactions between queries and documents, and hence achieve state of the art on many information retrieval benchmarks. However, their non-linear scoring function cannot be scaled to millions of documents, necessitating a three-stage process for inference: retrieving initial candidates via token retrieval, accessing all token vectors, and scoring the initial candidate documents. The non-linear scoring function is applied over all token vectors of each candidate document, making the inference process complicated and slow. In this paper, we aim to simplify the multi-vector retrieval by rethinking the role of token retrieval. We present XTR, ConteXtualized Token Retriever, which introduces a simple, yet novel, objective function that encourages the model to retrieve the most important document tokens first. The improvement to token retrieval allows XTR to rank candidates only using the retrieved t

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylee2023rethinking

Related papers

• Reducing The Footprint Of Multi-vector Retrieval With Minimal Performance Impact Via Token Pooling (2024)0.00
• CITADEL: Conditional Token Interaction Via Dynamic Lexical Routing For Efficient And Effective Multi-vector Retrieval (2022)13.05
• Multi-vector Retrieval As Sparse Alignment (2022)0.00
• SLIM: Sparsified Late Interaction For Multi-vector Retrieval With Inverted Indexes (2023)7.50
• Col-bandit: Zero-shot Query-time Pruning For Late-interaction Retrieval (2026)0.00
• Colbert-xm: A Modular Multi-vector Representation Model For Zero-shot Multilingual Information Retrieval (2024)0.00
• Reason To Contrast: A Cascaded Multimodal Retrieval Framework (2025)0.00
• Pylate: Flexible Training And Retrieval For Late Interaction Models (2025)3.58