Improving Mispronunciation Detection With Wav2vec2-based Momentum Pseudo-labeling For Accentedness And Intelligibility Assessment
2022 · Mu Yang, Kevin Hirschi, Stephen D. Looney, et al.
Abstract
Current leading mispronunciation detection and diagnosis (MDD) systems achieve promising performance via end-to-end phoneme recognition. One challenge of such end-to-end solutions is the scarcity of human-annotated phonemes on natural L2 speech. In this work, we leverage unlabeled L2 speech via a pseudo-labeling (PL) procedure and extend the fine-tuning approach based on pre-trained self-supervised learning (SSL) models. Specifically, we use Wav2vec 2.0 as our SSL model, and fine-tune it using original labeled L2 speech samples plus the created pseudo-labeled L2 speech samples. Our pseudo labels are dynamic and are produced by an ensemble of the online model on-the-fly, which ensures that our model is robust to pseudo label noise. We show that fine-tuning with pseudo labels achieves a 5.35% phoneme error rate reduction and 2.48% MDD F1 score improvement over a labeled-samples-only fine-tuning baseline. The proposed PL method is also shown to outperform conventional offline PL methods.
Authors
(none)
Tags
Stats
Related papers
- Improving End-to-end Modeling For Mispronunciation Detection With Effective Augmentation Mechanisms (2021)0.00
- A Full Text-dependent End To End Mispronunciation Detection And Diagnosis With Easy Data Augmentation Techniques (2021)0.00
- Advancing Momentum Pseudo-labeling With Conformer And Initialization Strategy (2021)6.34
- Multi-view Multi-task Representation Learning For Mispronunciation Detection (2023)0.00
- Speechblender: Speech Augmentation Framework For Mispronunciation Data Generation (2022)2.26
- Automatic Pronunciation Assessment Using Self-supervised Speech Representation Learning (2022)0.00
- Phonetic And Prosody-aware Self-supervised Learning Approach For Non-native Fluency Scoring (2023)3.58
- Adaptive Frequency Cepstral Coefficients For Word Mispronunciation Detection (2016)5.84