Multi-task Learning For End-to-end ASR Word And Utterance Confidence With Deletion Prediction

Abstract

Confidence scores are very useful for downstream applications of automatic speech recognition (ASR) systems. Recent works have proposed using neural networks to learn word or utterance confidence scores for end-to-end ASR. In those studies, word confidence by itself does not model deletions, and utterance confidence does not take advantage of word-level training signals. This paper proposes to jointly learn word confidence, word deletion, and utterance confidence. Empirical results show that multi-task learning with all three objectives improves confidence metrics (NCE, AUC, RMSE) without the need for increasing the model size of the confidence estimation module. Using the utterance-level confidence for rescoring also decreases the word error rates on Google's Voice Search and Long-tail Maps datasets by 3-5% relative, without needing a dedicated neural rescorer.

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyqiu2021multi

Related papers

• Confidence Estimation And Deletion Prediction Using Bidirectional Recurrent Neural Networks (2018)9.23
• An Evaluation Of Word-level Confidence Estimation For End-to-end Automatic Speech Recognition (2021)0.00
• Confidence Estimation For Attention-based Sequence-to-sequence Models For Speech Recognition (2020)11.49
• Accurate And Reliable Confidence Estimation Based On Non-autoregressive End-to-end Speech Recognition System (2023)4.52
• Sequence-level Confidence Classifier For ASR Utterance Accuracy And Application To Acoustic Models (2021)5.24
• Utterance-level Neural Confidence Measure For End-to-end Children Speech Recognition (2021)6.77
• Multi-task Language Modeling For Improving Speech Recognition Of Rare Words (2020)8.35
• 3M: Multi-loss, Multi-path And Multi-level Neural Networks For Speech Recognition (2022)8.67