Evaluating Stochastic Rankings With Expected Exposure

Abstract

We introduce the concept of *expected exposure* as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a *distribution over rankings* instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety

Stats

• citations128
• S2 citations—
• github stars0
• HF likes0
• heat score15.83
• arxiv keydiaz2020evaluating

Related papers

• Query Exposure Prediction For Groups Of Documents In Rankings (2024)4.52
• On The Calibration And Uncertainty Of Neural Learning To Rank Models (2021)0.00
• An Axiomatic Analysis Of Diversity Evaluation Metrics: Introducing The Rank-biased Utility Metric (2018)11.76
• Exposing Query Identification For Search Transparency (2021)6.77
• Not All Relevance Scores Are Equal: Efficient Uncertainty And Calibration Modeling For Deep Retrieval Models (2021)9.76
• Expertrank: A Multi-level Coarse-grained Expert-based Listwise Ranking Loss (2021)0.00
• Few-shot Prompting For Pairwise Ranking: An Effective Non-parametric Retrieval Model (2024)5.84
• Coverage, Not Averages: Semantic Stratification For Trustworthy Retrieval Evaluation (2026)0.00