Pre-training Transformer Decoder For End-to-end ASR Model With Unpaired Speech Data

Abstract

This paper studies a novel pre-training technique with unpaired speech data, Speech2C, for encoder-decoder based automatic speech recognition (ASR). Within a multi-task learning framework, we introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes, derived from an offline clustering model. One is to predict the pseudo codes via masked language modeling in encoder output, like HuBERT model, while the other lets the decoder learn to reconstruct pseudo codes autoregressively instead of generating textual scripts. In this way, the decoder learns to reconstruct original speech information with codes before learning to generate correct text. Comprehensive experiments on the LibriSpeech corpus show that the proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training, and also outperforms significantly the state-of-the-art wav2vec 2.0 and HuBERT on fine-tuning subsets of 10h and 100h.

Stats

• citations8
• S2 citations—
• github stars1439
• HF likes0
• heat score13.47
• arxiv keyao2022pre

Related papers

• Wav2seq: Pre-training Speech-to-text Encoder-decoder Models Using Pseudo Languages (2022)10.48
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019)0.00
• Unsupervised Pre-training For Sequence To Sequence Speech Recognition (2019)0.00
• Effective Decoder Masking For Transformer Based End-to-end Speech Recognition (2020)0.00
• Supervision-guided Codebooks For Masked Prediction In Speech Pre-training (2022)7.81
• Decoupling Pronunciation And Language For End-to-end Code-switching Automatic Speech Recognition (2020)0.00
• Improving Hybrid Ctc/attention End-to-end Speech Recognition With Pretrained Acoustic And Language Model (2021)8.82
• Guided Contrastive Self-supervised Pre-training For Automatic Speech Recognition (2022)0.00