Drowning In Documents: Consequences Of Scaling Reranker Inference

Abstract

Rerankers, typically cross-encoders, are computationally intensive but are frequently used because they are widely assumed to outperform cheaper initial IR systems. We challenge this assumption by measuring reranker performance for full retrieval, not just re-scoring first-stage retrieval. To provide a more robust evaluation, we prioritize strong first-stage retrieval using modern dense embeddings and test rerankers on a variety of carefully chosen, challenging tasks, including internally curated datasets to avoid contamination, and out-of-domain ones. Our empirical results reveal a surprising trend: the best existing rerankers provide initial improvements when scoring progressively more documents, but their effectiveness gradually declines and can even degrade quality beyond a certain limit. We hope that our findings will spur future research to improve reranking.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyjacob2024drowning

Related papers

• Refit: Relevance Feedback From A Reranker During Inference (2023)0.00
• CODER: An Efficient Framework For Improving Retrieval Through Contextual Document Embedding Reranking (2021)7.16
• Rank-k: Test-time Reasoning For Listwise Reranking (2025)0.00
• RRRA: Resampling And Reranking Through A Retriever Adapter (2025)0.00
• Towards Robust Ranker For Text Retrieval (2022)5.84
• Corank: Llm-based Compact Reranking With Document Features For Scientific Retrieval (2025)0.00
• SDR: Efficient Neural Re-ranking Using Succinct Document Representation (2021)3.58
• MICE: Minimal Interaction Cross-encoders For Efficient Re-ranking (2026)0.00