Evaluating User Perception Of Speech Recognition System Quality With Semantic Distance Metric

Abstract

Measuring automatic speech recognition (ASR) system quality is critical for creating user-satisfying voice-driven applications. Word Error Rate (WER) has been traditionally used to evaluate ASR system quality; however, it sometimes correlates poorly with user perception/judgement of transcription quality. This is because WER weighs every word equally and does not consider semantic correctness which has a higher impact on user perception. In this work, we propose evaluating ASR output hypotheses quality with SemDist that can measure semantic correctness by using the distance between the semantic vectors of the reference and hypothesis extracted from a pre-trained language model. Our experimental results of 71K and 36K user annotated ASR output quality show that SemDist achieves higher correlation with user perception than WER. We also show that SemDist has higher correlation with downstream Natural Language Understanding (NLU) tasks than WER.

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keykim2021evaluating

Related papers

• Semantic-wer: A Unified Metric For The Evaluation Of ASR Transcript For End Usability (2021)0.00
• A Reference-less Quality Metric For Automatic Speech Recognition Via Contrastive-learning Of A Multi-language Model With Self-supervision (2023)2.51
• H_eval: A New Hybrid Evaluation Metric For Automatic Speech Recognition Tasks (2022)6.34
• Automatic Speech Recognition System-independent Word Error Rate Estimation (2024)3.58
• Acoustics-guided Evaluation (AGE): A New Measure For Estimating Performance Of Speech Enhancement Algorithms For Robust ASR (2018)0.00
• ASR Error Management For Improving Spoken Language Understanding (2017)9.92
• Norefer: A Referenceless Quality Metric For Automatic Speech Recognition Via Semi-supervised Language Model Fine-tuning With Contrastive Learning (2023)0.00
• Speechbertscore: Reference-aware Automatic Evaluation Of Speech Generation Leveraging NLP Evaluation Metrics (2024)10.74