An Evaluation Of Word-level Confidence Estimation For End-to-end Automatic Speech Recognition

Abstract

Quantifying the confidence (or conversely the uncertainty) of a prediction is a highly desirable trait of an automatic system, as it improves the robustness and usefulness in downstream tasks. In this paper we investigate confidence estimation for end-to-end automatic speech recognition (ASR). Previous work has addressed confidence measures for lattice-based ASR, while current machine learning research mostly focuses on confidence measures for unstructured deep learning. However, as the ASR systems are increasingly being built upon deep end-to-end methods, there is little work that tries to develop confidence measures in this context. We fill this gap by providing an extensive benchmark of popular confidence methods on four well-known speech datasets. There are two challenges we overcome in adapting existing methods: working on structured data (sequences) and obtaining confidences at a coarser level than the predictions (words instead of tokens). Our results suggest that a strong basel

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyoneata2021an

Related papers

• Confidence Estimation For Attention-based Sequence-to-sequence Models For Speech Recognition (2020)11.49
• Confidence Estimation For Black Box Automatic Speech Recognition Systems Using Lattice Recurrent Neural Networks (2019)8.35
• Accurate And Reliable Confidence Estimation Based On Non-autoregressive End-to-end Speech Recognition System (2023)4.52
• Fast Entropy-based Methods Of Word-level Confidence Estimation For End-to-end Automatic Speech Recognition (2022)7.16
• Utterance-level Neural Confidence Measure For End-to-end Children Speech Recognition (2021)6.77
• Multi-task Learning For End-to-end ASR Word And Utterance Confidence With Deletion Prediction (2021)7.50
• Unsupervised Uncertainty Measures Of Automatic Speech Recognition For Non-intrusive Speech Intelligibility Prediction (2022)6.77
• Teles: Temporal Lexeme Similarity Score To Estimate Confidence In End-to-end ASR (2024)6.34