Advancing Momentum Pseudo-labeling With Conformer And Initialization Strategy

Abstract

Pseudo-labeling (PL), a semi-supervised learning (SSL) method where a seed model performs self-training using pseudo-labels generated from untranscribed speech, has been shown to enhance the performance of end-to-end automatic speech recognition (ASR). Our prior work proposed momentum pseudo-labeling (MPL), which performs PL-based SSL via an interaction between online and offline models, inspired by the mean teacher framework. MPL achieves remarkable results on various semi-supervised settings, showing robustness to variations in the amount of data and domain mismatch severity. However, there is further room for improving the seed model used to initialize the MPL training, as it is in general critical for a PL-based method to start training from high-quality pseudo-labels. To this end, we propose to enhance MPL by (1) introducing the Conformer architecture to boost the overall recognition accuracy and (2) exploiting iterative pseudo-labeling with a language model to improve the seed mo

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyhiguchi2021advancing

Related papers

• Intermpl: Momentum Pseudo-labeling With Intermediate CTC Loss (2022)0.00
• Improving Mispronunciation Detection With Wav2vec2-based Momentum Pseudo-labeling For Accentedness And Intelligibility Assessment (2022)7.16
• Slimipl: Language-model-free Iterative Pseudo-labeling (2020)10.74
• Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Mask (2021)5.84
• An Adapter Based Multi-label Pre-training For Speech Separation And Enhancement (2022)7.50
• Self-training For End-to-end Speech Recognition (2019)15.48
• Alternative Pseudo-labeling For Semi-supervised Automatic Speech Recognition (2023)10.48
• Multi-task Pseudo-label Learning For Non-intrusive Speech Quality Assessment Model (2023)0.00