Speechcolab Leaderboard: An Open-source Platform For Automatic Speech Recognition Evaluation

Abstract

In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, the impartial and replicable evaluation of these ASR systems encounters challenges due to various crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a general-purpose, open-source platform designed for ASR evaluation. With this platform: (i) We report a comprehensive benchmark, unveiling the current state-of-the-art panorama for ASR systems, covering both open-source models and industrial commercial services. (ii) We quantize how distinct nuances in the scoring pipeline influence the final benchmark outcomes. These include nuances related to capitalization, punctuation, interjection, contraction, synonym usage, compound words, etc. These issues have gained prominence in the context of

Stats

• citations2
• S2 citations—
• github stars545
• HF likes0
• heat score9.05
• arxiv keydu2024speechcolab

Related papers

• Open ASR Leaderboard: Towards Reproducible And Transparent Multilingual And Long-form Speech Recognition Evaluation (2025)0.00
• Framework For Curating Speech Datasets And Evaluating ASR Systems: A Case Study For Polish (2024)2.16
• ASR-GLUE: A New Multi-task Benchmark For Asr-robust Natural Language Understanding (2021)0.00
• Lebenchmark: A Reproducible Framework For Assessing Self-supervised Representation Learning From Speech (2021)11.39
• Speechrole: A Large-scale Dataset And Benchmark For Evaluating Speech Role-playing Agents (2025)1.91
• Toward Practical Automatic Speech Recognition And Post-processing: A Call For Explainable Error Benchmark Guideline (2024)0.00
• English Accent Accuracy Analysis In A State-of-the-art Automatic Speech Recognition System (2021)0.00
• Accented Speech Recognition: Benchmarking, Pre-training, And Diverse Data (2022)0.00