On The Calibration And Uncertainty Of Neural Learning To Rank Models

Abstract

According to the Probability Ranking Principle (PRP), ranking documents in decreasing order of their probability of relevance leads to an optimal document ranking for ad-hoc retrieval. The PRP holds when two conditions are met: [C1] the models are well calibrated, and, [C2] the probabilities of relevance are reported with certainty. We know however that deep neural networks (DNNs) are often not well calibrated and have several sources of uncertainty, and thus [C1] and [C2] might not be satisfied by neural rankers. Given the success of neural Learning to Rank (L2R) approaches-and here, especially BERT-based approaches-we first analyze under which circumstances deterministic, i.e. outputs point estimates, neural rankers are calibrated. Then, motivated by our findings we use two techniques to model the uncertainty of neural rankers leading to the proposed stochastic rankers, which output a predictive distribution of relevance as opposed to point estimates. Our experimental results on the

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keypenha2021on

Related papers

• Not All Relevance Scores Are Equal: Efficient Uncertainty And Calibration Modeling For Deep Retrieval Models (2021)9.76
• A Deep Look Into Neural Ranking Models For Information Retrieval (2019)17.73
• Towards Axiomatic Explanations For Neural Ranking Models (2021)10.35
• Scalable Exploration For Neural Online Learning To Rank With Perturbed Feedback (2022)0.00
• Neural Ranking Models For Document Retrieval (2021)11.08
• Enhancing The Ranking Context Of Dense Retrieval Methods Through Reciprocal Nearest Neighbors (2023)4.52
• Setrank: Learning A Permutation-invariant Ranking Model For Information Retrieval (2019)0.00
• Neural Ranking Models With Weak Supervision (2017)17.02