Multitask Learning And Joint Optimization For Transformer-rnn-transducer Speech Recognition

Abstract

Recently, several types of end-to-end speech recognition methods named transformer-transducer were introduced. According to those kinds of methods, transcription networks are generally modeled by transformer-based neural networks, while prediction networks could be modeled by either transformers or recurrent neural networks (RNN). This paper explores multitask learning, joint optimization, and joint decoding methods for transformer-RNN-transducer systems. Our proposed methods have the main advantage in that the model can maintain information on the large text corpus. We prove their effectiveness by performing experiments utilizing the well-known ESPNET toolkit for the widely used Librispeech datasets. We also show that the proposed methods can reduce word error rate (WER) by 16.6 % and 13.3 % for test-clean and test-other datasets, respectively, without changing the overall model structure nor exploiting an external LM.

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keyjeon2020multitask

Related papers

• Transformer-transducer: End-to-end Speech Recognition With Self-attention (2019)0.00
• Improving RNN Transducer Modeling For End-to-end Speech Recognition (2019)0.00
• Transformer Transducer: A Streamable Speech Recognition Model With Transformer Encoders And RNN-T Loss (2020)18.58
• Multitask Training With Text Data For End-to-end Speech Recognition (2020)7.50
• Research On Modeling Units Of Transformer Transducer For Mandarin Speech Recognition (2020)0.00
• Exploring Architectures, Data And Units For Streaming End-to-end Speech Recognition With Rnn-transducer (2018)16.21
• Improved Neural Language Model Fusion For Streaming Recurrent Neural Network Transducer (2020)8.82
• Developing Real-time Streaming Transformer Transducer For Speech Recognition On Large-scale Dataset (2020)0.00