Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?

Abstract

Neural retrieval models are generally regarded as fundamentally different from the retrieval techniques used in the late 1990's when the TREC ad hoc test collections were constructed. They thus provide the opportunity to empirically test the claim that pooling-built test collections can reliably evaluate retrieval systems that did not contribute to the construction of the collection (in other words, that such collections can be reusable). To test the reusability claim, we asked TREC assessors to judge new pools created from new search results for the TREC-8 ad hoc collection. These new search results consisted of five new runs (one each from three transformer-based models and two baseline runs that use BM25) plus the set of TREC-8 submissions that did not previously contribute to pools. The new runs did retrieve previously unseen documents, but the vast majority of those documents were not relevant. The ranking of all runs by mean evaluation score when evaluated using the official TREC

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvoorhees2022can

Related papers

• Evaluating Interpolation And Extrapolation Performance Of Neural Retrieval Models (2022)6.77
• Systematic Evaluation Of Neural Retrieval Models On The Touch\'e 2020 Argument Retrieval Subset Of BEIR (2024)9.31
• Evaluating Dense Passage Retrieval Using Transformers (2022)0.00
• How Different Are Pre-trained Transformers For Text Ranking? (2022)7.81
• How Train-test Leakage Affects Zero-shot Retrieval (2022)3.58
• Noisy Self-training With Synthetic Queries For Dense Retrieval (2023)0.00
• A Deep Look Into Neural Ranking Models For Information Retrieval (2019)17.73
• Improving Neural Ranking Models With Traditional IR Methods (2023)0.00