Improved Neural Language Model Fusion For Streaming Recurrent Neural Network Transducer

Abstract

Recurrent Neural Network Transducer (RNN-T), like most end-to-end speech recognition model architectures, has an implicit neural network language model (NNLM) and cannot easily leverage unpaired text data during training. Previous work has proposed various fusion methods to incorporate external NNLMs into end-to-end ASR to address this weakness. In this paper, we propose extensions to these techniques that allow RNN-T to exploit external NNLMs during both training and inference time, resulting in 13-18% relative Word Error Rate improvement on Librispeech compared to strong baselines. Furthermore, our methods do not incur extra algorithmic latency and allow for flexible plug-and-play of different NNLMs without re-training. We also share in-depth analysis to better understand the benefits of the different NNLM fusion methods. Our work provides a reliable technique for leveraging unpaired text data to significantly improve RNN-T while keeping the system streamable, flexible, and lightweig

Stats

• citations14
• S2 citations—
• github stars0
• HF likes0
• heat score8.82
• arxiv keykim2020improved

Related papers

• Alignment Restricted Streaming Recurrent Neural Network Transducer (2020)11.19
• Deep Shallow Fusion For RNN-T Personalization (2020)12.81
• On Language Model Integration For RNN Transducer Based Speech Recognition (2021)9.59
• Integrating Text Inputs For Training And Adapting RNN Transducer ASR Models (2022)9.59
• Exploring Architectures, Data And Units For Streaming End-to-end Speech Recognition With Rnn-transducer (2018)16.21
• Improving RNN Transducer Modeling For End-to-end Speech Recognition (2019)0.00
• Transformer Transducer: A Streamable Speech Recognition Model With Transformer Encoders And RNN-T Loss (2020)18.58
• Transducer-llama: Integrating Llms Into Streamable Transducer-based Speech Recognition (2024)3.58