Multi-task Language Modeling For Improving Speech Recognition Of Rare Words

Abstract

End-to-end automatic speech recognition (ASR) systems are increasingly popular due to their relative architectural simplicity and competitive performance. However, even though the average accuracy of these systems may be high, the performance on rare content words often lags behind hybrid ASR systems. To address this problem, second-pass rescoring is often applied leveraging upon language modeling. In this paper, we propose a second-pass system with multi-task learning, utilizing semantic targets (such as intent and slot prediction) to improve speech recognition performance. We show that our rescoring model trained with these additional tasks outperforms the baseline rescoring model, trained with only the language modeling task, by 1.4% on a general test and by 2.6% on a rare word test set in terms of word-error-rate relative (WERR). Our best ASR system with multi-task LM shows 4.6% WERR deduction compared with RNN Transducer only ASR baseline for rare words recognition.

Stats

• citations12
• S2 citations—
• github stars0
• HF likes0
• heat score8.35
• arxiv keyyang2020multi

Related papers

• Audio-attention Discriminative Language Model For ASR Rescoring (2019)9.23
• Discriminative Speech Recognition Rescoring With Pre-trained Language Models (2023)2.26
• End-to-end Speech Recognition With Word-based RNN Language Models (2018)0.00
• Multi-stage Large Language Model Correction For Speech Recognition (2023)0.00
• E2e-based Multi-task Learning Approach To Joint Speech And Accent Recognition (2021)0.00
• End-to-end Contextual Speech Recognition Using Class Language Models And A Token Passing Decoder (2018)11.08
• Memory Augmented Lookup Dictionary Based Language Modeling For Automatic Speech Recognition (2022)0.00
• Integrating Pre-trained Speech And Language Models For End-to-end Speech Recognition (2023)0.00