Transformer Language Models With Lstm-based Cross-utterance Information Representation

Abstract

The effective incorporation of cross-utterance information has the potential to improve language models (LMs) for automatic speech recognition (ASR). To extract more powerful and robust cross-utterance representations for the Transformer LM (TLM), this paper proposes the R-TLM which uses hidden states in a long short-term memory (LSTM) LM. To encode the cross-utterance information, the R-TLM incorporates an LSTM module together with a segment-wise recurrence in some of the Transformer blocks. In addition to the LSTM module output, a shortcut connection using a fusion layer that bypasses the LSTM module is also investigated. The proposed system was evaluated on the AMI meeting corpus, the Eval2000 and the RT03 telephone conversation evaluation sets. The best R-TLM achieved 0.9%, 0.6%, and 0.8% absolute WER reductions over the single-utterance TLM baseline, and 0.5%, 0.3%, 0.2% absolute WER reductions over a strong cross-utterance TLM baseline on the AMI evaluation set, Eval2000 and RT03

Stats

• citations24
• S2 citations—
• github stars0
• HF likes0
• heat score10.48
• arxiv keysun2021transformer

Related papers

• Memory Augmented Lookup Dictionary Based Language Modeling For Automatic Speech Recognition (2022)0.00
• Improved Neural Language Model Fusion For Streaming Recurrent Neural Network Transducer (2020)8.82
• Transducer-llama: Integrating Llms Into Streamable Transducer-based Speech Recognition (2024)3.58
• On Language Model Integration For RNN Transducer Based Speech Recognition (2021)9.59
• Integrating Text Inputs For Training And Adapting RNN Transducer ASR Models (2022)9.59
• Transformer In Action: A Comparative Study Of Transformer-based Acoustic Models For Large Scale Speech Recognition Applications (2020)9.41
• Effective Cross-utterance Language Modeling For Conversational Speech Recognition (2021)2.26
• Latent Speech-text Transformer (2025)3.04