Cross-modal ASR Post-processing System For Error Correction And Utterance Rejection

Abstract

Although modern automatic speech recognition (ASR) systems can achieve high performance, they may produce errors that weaken readers' experience and do harm to downstream tasks. To improve the accuracy and reliability of ASR hypotheses, we propose a cross-modal post-processing system for speech recognizers, which 1) fuses acoustic features and textual features from different modalities, 2) joints a confidence estimator and an error corrector in multi-task learning fashion and 3) unifies error correction and utterance rejection modules. Compared with single-modal or single-task models, our proposed system is proved to be more effective and efficient. Experiment result shows that our post-processing system leads to more than 10% relative reduction of character error rate (CER) for both single-speaker and multi-speaker speech on our industrial ASR system, with about 1.7ms latency for each token, which ensures that extra latency introduced by post-processing is acceptable in streaming spee

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydu2022cross

Related papers

• Ed-cec: Improving Rare Word Recognition Using Asr Postprocessing Based On Error Detection And Context-aware Error Correction (2023)6.34
• Patcorrect: Non-autoregressive Phoneme-augmented Transformer For ASR Error Correction (2023)0.00
• Speech Emotion Recognition With ASR Transcripts: A Comprehensive Study On Word Error Rate And Fusion Techniques (2024)9.03
• ASR Error Management For Improving Spoken Language Understanding (2017)9.92
• MF-AED-AEC: Speech Emotion Recognition By Leveraging Multimodal Fusion, Asr Error Detection, And Asr Error Correction (2024)0.00
• MLCA-AVSR: Multi-layer Cross Attention Fusion Based Audio-visual Speech Recognition (2024)10.07
• Elevating Robust Multi-talker ASR By Decoupling Speaker Separation And Speech Recognition (2025)0.00
• Whispering Llama: A Cross-modal Generative Error Correction Framework For Speech Recognition (2023)16.15