Improving RNN-T ASR Accuracy Using Context Audio

Abstract

We present a training scheme for streaming automatic speech recognition (ASR) based on recurrent neural network transducers (RNN-T) which allows the encoder network to learn to exploit context audio from a stream, using segmented or partially labeled sequences of the stream during training. We show that the use of context audio during training and inference can lead to word error rate reductions of more than 6% in a realistic production setting for a voice assistant ASR system. We investigate the effect of the proposed training approach on acoustically challenging data containing background speech and present data points which indicate that this approach helps the network learn both speaker and environment adaptation. To gain further insight into the ability of a long short-term memory (LSTM) based ASR encoder to exploit long-term context, we also visualize RNN-T loss gradients with respect to the input.

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyschwarz2020improving

Related papers

• Improving RNN Transducer Based ASR With Auxiliary Tasks (2020)9.59
• Leveraging Acoustic Contextual Representation By Audio-textual Cross-modal Learning For Conversational ASR (2022)0.00
• Semi-autoregressive Streaming ASR With Label Context (2023)0.00
• Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers (2021)10.07
• Training ASR Models By Generation Of Contextual Information (2019)0.00
• Alignment Restricted Streaming Recurrent Neural Network Transducer (2020)11.19
• Integrating Text Inputs For Training And Adapting RNN Transducer ASR Models (2022)9.59
• Towards Effective And Compact Contextual Representation For Conformer Transducer Speech Recognition Systems (2023)7.16