CODER: An Efficient Framework For Improving Retrieval Through Contextual Document Embedding Reranking

Abstract

Contrastive learning has been the dominant approach to training dense retrieval models. In this work, we investigate the impact of ranking context - an often overlooked aspect of learning dense retrieval models. In particular, we examine the effect of its constituent parts: jointly scoring a large number of negatives per query, using retrieved (query-specific) instead of random negatives, and a fully list-wise loss. To incorporate these factors into training, we introduce Contextual Document Embedding Reranking (CODER), a highly efficient retrieval framework. When reranking, it incurs only a negligible computational overhead on top of a first-stage method at run time (delay per query in the order of milliseconds), allowing it to be easily combined with any state-of-the-art dual encoder method. After fine-tuning through CODER, which is a lightweight and fast process, models can also be used as stand-alone retrievers. Evaluating CODER in a large set of experiments on the MS~MARCO and Tri

Stats

• citations8
• S2 citations—
• github stars0
• HF likes0
• heat score7.16
• arxiv keyzerveas2021coder

Related papers

• Noise-robust Dense Retrieval Via Contrastive Alignment Post Training (2023)0.00
• Enhancing The Ranking Context Of Dense Retrieval Methods Through Reciprocal Nearest Neighbors (2023)4.52
• Drowning In Documents: Consequences Of Scaling Reranker Inference (2024)0.00
• Rare: Retrieval Augmented Retrieval With In-context Examples (2024)0.00
• Efficient Document Ranking With Learnable Late Interactions (2024)0.00
• Comparing Neighbors Together Makes It Easy: Jointly Comparing Multiple Candidates For Efficient And Effective Retrieval (2024)4.52
• Refit: Relevance Feedback From A Reranker During Inference (2023)0.00
• Efficient Neural Ranking Using Forward Indexes And Lightweight Encoders (2023)5.24