Modular Hybrid Autoregressive Transducer

Abstract

Text-only adaptation of a transducer model remains challenging for end-to-end speech recognition since the transducer has no clearly separated acoustic model (AM), language model (LM) or blank model. In this work, we propose a modular hybrid autoregressive transducer (MHAT) that has structurally separated label and blank decoders to predict label and blank distributions, respectively, along with a shared acoustic encoder. The encoder and label decoder outputs are directly projected to AM and internal LM scores and then added to compute label posteriors. We train MHAT with an internal LM loss and a HAT loss to ensure that its internal LM becomes a standalone neural LM that can be effectively adapted to text. Moreover, text adaptation of MHAT fosters a much better LM fusion than internal LM subtraction-based methods. On Google's large-scale production data, a multi-domain MHAT adapted with 100B sentences achieves relative WER reductions of up to 12.4% without LM fusion and 21.5% with LM

Stats

• citations12
• S2 citations—
• github stars0
• HF likes0
• heat score8.35
• arxiv keymeng2022modular

Related papers

• Boosting Hybrid Autoregressive Transducer-based ASR With Internal Acoustic Model Training And Dual Blank Thresholding (2024)2.26
• On Minimum Word Error Rate Training Of The Hybrid Autoregressive Transducer (2020)4.52
• Hybrid Attention-based Encoder-decoder Model For Efficient Language Model Adaptation (2023)0.00
• Integrating Text Inputs For Training And Adapting RNN Transducer ASR Models (2022)9.59
• Label-synchronous Neural Transducer For Adaptable Online E2E Speech Recognition (2023)3.58
• Transformer-based Acoustic Modeling For Hybrid Speech Recognition (2019)16.30
• Phoneme Based Neural Transducer For Large Vocabulary Speech Recognition (2020)9.59
• Unsupervised Model-based Speaker Adaptation Of End-to-end Lattice-free MMI Model For Speech Recognition (2022)2.26