Performance Monitoring For End-to-end Speech Recognition

Abstract

Measuring performance of an automatic speech recognition (ASR) system without ground-truth could be beneficial in many scenarios, especially with data from unseen domains, where performance can be highly inconsistent. In conventional ASR systems, several performance monitoring (PM) techniques have been well-developed to monitor performance by looking at tri-phone posteriors or pre-softmax activations from neural network acoustic modeling. However, strategies for monitoring more recently developed end-to-end ASR systems have not yet been explored, and so that is the focus of this paper. We adapt previous PM measures (Entropy, M-measure and Auto-encoder) and apply our proposed RNN predictor in the end-to-end setting. These measures utilize the decoder output layer and attention probability vectors, and their predictive power is measured with simple linear models. Our findings suggest that decoder-level features are more feasible and informative than attention-level probabilities for PM m

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyli2019performance

Related papers

• A Comparison Of End-to-end Models For Long-form Speech Recognition (2019)12.93
• Audio-attention Discriminative Language Model For ASR Rescoring (2019)9.23
• Unsupervised Uncertainty Measures Of Automatic Speech Recognition For Non-intrusive Speech Intelligibility Prediction (2022)6.77
• Acoustics-guided Evaluation (AGE): A New Measure For Estimating Performance Of Speech Enhancement Algorithms For Robust ASR (2018)0.00
• Utterance-level Neural Confidence Measure For End-to-end Children Speech Recognition (2021)6.77
• Phonetic-assisted Multi-target Units Modeling For Improving Conformer-transducer ASR System (2022)4.52
• Multi-task Language Modeling For Improving Speech Recognition Of Rare Words (2020)8.35
• Towards A Competitive End-to-end Speech Recognition For Chime-6 Dinner Party Transcription (2020)6.77