The Overlooked Role Of Graded Relevance Thresholds In Multilingual Dense Retrieval

Abstract

Dense retrieval models are typically fine-tuned with contrastive learning objectives that require binary relevance judgments, even though relevance is inherently graded. We analyze how graded relevance scores and the threshold used to convert them into binary labels affect multilingual dense retrieval. Using a multilingual dataset with LLM-annotated relevance scores, we examine monolingual, multilingual mixture, and cross-lingual retrieval scenarios. Our findings show that the optimal threshold varies systematically across languages and tasks, often reflecting differences in resource level. A well-chosen threshold can improve effectiveness, reduce the amount of fine-tuning data required, and mitigate annotation noise, whereas a poorly chosen one can degrade performance. We argue that graded relevance is a valuable but underutilized signal for dense retrieval, and that threshold calibration should be treated as a principled component of the fine-tuning pipeline.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywullach2026the

Related papers

• Bixse: Improving Dense Retrieval Via Probabilistic Graded Relevance Distillation (2025)0.00
• Enhancing The Ranking Context Of Dense Retrieval Methods Through Reciprocal Nearest Neighbors (2023)4.52
• What Drives Cross-lingual Ranking? Retrieval Approaches With Multilingual Language Models (2025)0.00
• Formalized Information Needs Improve Large-language-model Relevance Judgments (2026)0.00
• Boosting Data Utilization For Multilingual Dense Retrieval (2025)0.00
• Supervised Fine-tuning Or Contrastive Learning? Towards Better Multimodal LLM Reranking (2025)0.00
• Generative Retrieval Meets Multi-graded Relevance (2024)2.26
• Mine And Refine: Optimizing Graded Relevance In E-commerce Search Retrieval (2026)0.00