Evaluating Dense Passage Retrieval Using Transformers

Abstract

Although representational retrieval models based on Transformers have been able to make major advances in the past few years, and despite the widely accepted conventions and best-practices for testing such models, a \(\textit\{standardized\}\) evaluation framework for testing them has not been developed. In this work, we formalize the best practices and conventions followed by researchers in the literature, paving the path for more standardized evaluations - and therefore more fair comparisons between the models. Our framework (1) embeds the documents and queries; (2) for each query-document pair, computes the relevance score based on the dot product of the document and query embedding; (3) uses the \(\texttt\{dev\}\) set of the MSMARCO dataset to evaluate the models; (4) uses the \(\texttt\{trec_eval\}\) script to calculate MRR@100, which is the primary metric used to evaluate the models. Most importantly, we showcase the use of this framework by experimenting on some of the most well

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysadri2022evaluating

Related papers

• Training Vision Transformers For Image Retrieval (2021)0.00
• How Different Are Pre-trained Transformers For Text Ranking? (2022)7.81
• Thinking Fast And Slow: Efficient Text-to-visual Retrieval With Transformers (2021)15.16
• Shallow Cross-encoders For Low-latency Retrieval (2024)2.26
• Boosting Vision Transformers For Image Retrieval (2022)15.28
• How Does Generative Retrieval Scale To Millions Of Passages? (2023)10.61
• TPRF: A Transformer-based Pseudo-relevance Feedback Model For Efficient And Effective Retrieval (2024)0.00
• Scalable And Effective Generative Information Retrieval (2023)10.48