Semantic-wer: A Unified Metric For The Evaluation Of ASR Transcript For End Usability

Abstract

Recent advances in supervised, semi-supervised and self-supervised deep learning algorithms have shown significant improvement in the performance of automatic speech recognition(ASR) systems. The state-of-the-art systems have achieved a word error rate (WER) less than 5%. However, in the past, researchers have argued the non-suitability of the WER metric for the evaluation of ASR systems for downstream tasks such as spoken language understanding (SLU) and information retrieval. The reason is that the WER works at the surface level and does not include any syntactic and semantic knowledge.The current work proposes Semantic-WER (SWER), a metric to evaluate the ASR transcripts for downstream applications in general. The SWER can be easily customized for any down-stream task.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyroy2021semantic

Related papers

• Evaluating User Perception Of Speech Recognition System Quality With Semantic Distance Metric (2021)6.77
• Automatic Speech Recognition System-independent Word Error Rate Estimation (2024)3.58
• Beyond Levenshtein: Leveraging Multiple Algorithms For Robust Word Error Rate Computations And Granular Error Classifications (2024)2.26
• H_eval: A New Hybrid Evaluation Metric For Automatic Speech Recognition Tasks (2022)6.34
• Fast Word Error Rate Estimation Using Self-supervised Representations For Speech And Text (2023)5.24
• On The Impact Of Word Error Rate On Acoustic-linguistic Speech Emotion Recognition: An Update For The Deep Learning Era (2021)0.00
• A Reference-less Quality Metric For Automatic Speech Recognition Via Contrastive-learning Of A Multi-language Model With Self-supervision (2023)2.51
• On Word Error Rate Definitions And Their Efficient Computation For Multi-speaker Speech Recognition Systems (2022)9.76