Do Neural Ranking Models Intensify Gender Bias?

Abstract

Concerns regarding the footprint of societal biases in information retrieval (IR) systems have been raised in several previous studies. In this work, we examine various recent IR models from the perspective of the degree of gender bias in their retrieval results. To this end, we first provide a bias measurement framework which includes two metrics to quantify the degree of the unbalanced presence of gender-related concepts in a given IR model's ranking list. To examine IR models by means of the framework, we create a dataset of non-gendered queries, selected by human annotators. Applying these queries to the MS MARCO Passage retrieval collection, we then measure the gender bias of a BM25 model and several recent neural ranking models. The results show that while all models are strongly biased toward male, the neural models, and in particular the ones based on contextualized embedding models, significantly intensify gender bias. Our experiments also show an overall increase in the gende

Stats

• citations45
• S2 citations—
• github stars0
• HF likes0
• heat score12.47
• arxiv keyrekabsaz2020do

Related papers

• Debiasing Gender Bias In Information Retrieval Models (2022)0.00
• A Deep Look Into Neural Ranking Models For Information Retrieval (2019)17.73
• An Empirical Study Of Position Bias In Modern Information Retrieval (2025)1.69
• Mitigating Test-time Bias For Fair Image Retrieval (2023)0.00
• Writing Style Matters: An Examination Of Bias And Fairness In Information Retrieval Systems (2024)4.52
• Neural Models For Information Retrieval (2017)0.00
• Invisible Relevance Bias: Text-image Retrieval Models Prefer Ai-generated Images (2023)9.23
• Ms-shift: An Analysis Of MS MARCO Distribution Shifts On Neural Retrieval (2022)4.52