How Different Are Pre-trained Transformers For Text Ranking?

Abstract

In recent years, large pre-trained transformers have led to substantial gains in performance over traditional retrieval models and feedback approaches. However, these results are primarily based on the MS Marco/TREC Deep Learning Track setup, with its very particular setup, and our understanding of why and how these models work better is fragmented at best. We analyze effective BERT-based cross-encoders versus traditional BM25 ranking for the passage retrieval task where the largest gains have been observed, and investigate two main questions. On the one hand, what is similar? To what extent does the neural ranker already encompass the capacity of traditional rankers? Is the gain in performance due to a better ranking of the same documents (prioritizing precision)? On the other hand, what is different? Can it retrieve effectively documents missed by traditional systems (prioritizing recall)? We discover substantial differences in the notion of relevance identifying strengths and weakne

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keyrau2022how

Related papers

• Injecting The BM25 Score As Text Improves Bert-based Re-rankers (2023)10.48
• On The Interpolation Of Contextualized Term-based Ranking With BM25 For Query-by-example Retrieval (2022)7.50
• Shallow Cross-encoders For Low-latency Retrieval (2024)2.26
• Improving Neural Ranking Models With Traditional IR Methods (2023)0.00
• Transfer Learning Approaches For Building Cross-language Dense Retrieval Models (2022)10.97
• Evaluating Dense Passage Retrieval Using Transformers (2022)0.00
• Comparative Analysis Of Lion And Adamw Optimizers For Cross-encoder Reranking With Minilm, GTE, And Modernbert (2025)0.00
• Quality And Cost Trade-offs In Passage Re-ranking Task (2021)0.00