Hindi-beir : A Large Scale Retrieval Benchmark In Hindi

Abstract

Given the large number of Hindi speakers worldwide, there is a pressing need for robust and efficient information retrieval systems for Hindi. Despite ongoing research, there is a lack of comprehensive benchmark for evaluating retrieval models in Hindi. To address this gap, we introduce the Hindi version of the BEIR benchmark, which includes a subset of English BEIR datasets translated to Hindi, existing Hindi retrieval datasets, and synthetically created datasets for retrieval. The benchmark is comprised of \(15\) datasets spanning across \(8\) distinct tasks. We evaluate state-of-the-art multilingual retrieval models on this benchmark to identify task and domain-specific challenges and their impact on retrieval performance. By releasing this benchmark and a set of relevant baselines, we enable researchers to understand the limitations and capabilities of current Hindi retrieval models, promoting advancements in this critical area. The datasets from Hindi-BEIR are publicly available.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyacharya2024hindi

Related papers

• Benchmarking And Building Zero-shot Hindi Retrieval Model With Hindi-beir And NLLB-E5 (2024)0.00
• BEIR: A Heterogenous Benchmark For Zero-shot Evaluation Of Information Retrieval Models (2021)6.67
• Resources For Brewing BEIR: Reproducible Reference Models And An Official Leaderboard (2023)0.00
• Mfollowir: A Multilingual Benchmark For Instruction Following In Retrieval (2025)0.00
• Neuclirbench: A Modern Evaluation Collection For Monolingual, Cross-language, And Multilingual Information Retrieval (2025)0.00
• MAIR: A Massive Benchmark For Evaluating Instructed Retrieval (2024)6.41
• Uniir: Training And Benchmarking Universal Multimodal Information Retrievers (2023)10.48
• Posir: Position-aware Heterogeneous Information Retrieval Benchmark (2026)0.00