Evaluation Measures For Relevance And Credibility In Ranked Lists

Abstract

Recent discussions on alternative facts, fake news, and post truth politics have motivated research on creating technologies that allow people not only to access information, but also to assess the credibility of the information presented to them by information retrieval systems. Whereas technology is in place for filtering information according to relevance and/or credibility, no single measure currently exists for evaluating the accuracy or precision (and more generally effectiveness) of both the relevance and the credibility of retrieved results. One obvious way of doing so is to measure relevance and credibility effectiveness separately, and then consolidate the two measures into one. There at least two problems with such an approach: (I) it is not certain that the same criteria are applied to the evaluation of both relevance and credibility (and applying different criteria introduces bias to the evaluation); (II) many more and richer measures exist for assessing relevance effectiv

Stats

• citations20
• S2 citations—
• github stars0
• HF likes0
• heat score9.92
• arxiv keylioma2017evaluation

Related papers

• Offline Evaluation Measures Of Fairness In Recommender Systems (2026)0.00
• On The Effect Of Ranking Axioms On IR Evaluation Metrics (2022)4.52
• Enhancing Documents With Multidimensional Relevance Statements In Cross-encoder Re-ranking (2023)0.00
• An Intrinsic Framework Of Information Retrieval Evaluation Measures (2023)2.26
• An Axiomatic Analysis Of Diversity Evaluation Metrics: Introducing The Rank-biased Utility Metric (2018)11.76
• Not All Relevance Scores Are Equal: Efficient Uncertainty And Calibration Modeling For Deep Retrieval Models (2021)9.76
• A Deep Look Into Neural Ranking Models For Information Retrieval (2019)17.73
• Formalized Information Needs Improve Large-language-model Relevance Judgments (2026)0.00