Mockingjay: Unsupervised Speech Representation Learning With Deep Bidirectional Transformer Encoders

Abstract

We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech. Previous speech representation methods learn through conditioning on past frames and predicting information about future frames. Whereas Mockingjay is designed to predict the current frame through jointly conditioning on both past and future contexts. The Mockingjay representation improves performance for a wide range of downstream tasks, including phoneme classification, speaker recognition, and sentiment classification on spoken content, while outperforming other approaches. Mockingjay is empirically powerful and can be fine-tuned with downstream models, with only 2 epochs we further improve performance dramatically. In a low resource setting with only 0.1% of labeled data, we outperform the result of Mel-features that uses all 100% labeled data.

Stats

• citations199
• S2 citations—
• github stars0
• HF likes0
• heat score17.26
• arxiv keyliu2019mockingjay

Related papers

• Unsupervised Pre-training Of Bidirectional Speech Encoders Via Masked Reconstruction (2020)12.33
• An Unsupervised Autoregressive Model For Speech Representation Learning (2019)17.26
• Self-supervised Rewiring Of Pre-trained Speech Encoders: Towards Faster Fine-tuning With Less Labels In Speech Processing (2022)3.58
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019)0.00
• Bidirectional Representations For Low Resource Spoken Language Understanding (2022)0.00
• Unispeech: Unified Speech Representation Learning With Labeled And Unlabeled Data (2021)0.00
• Learning Representations Of Emotional Speech With Deep Convolutional Generative Adversarial Networks (2017)0.00
• Towards Unsupervised Speech Recognition And Synthesis With Quantized Speech Representation Learning (2019)0.00