Coca-mdd: A Coupled Cross-attention Based Framework For Streaming Mispronunciation Detection And Diagnosis

Abstract

Mispronunciation detection and diagnosis (MDD) is a popular research focus in computer-aided pronunciation training (CAPT) systems. End-to-end (e2e) approaches are becoming dominant in MDD. However an e2e MDD model usually requires entire speech utterances as input context, which leads to significant time latency especially for long paragraphs. We propose a streaming e2e MDD model called CoCA-MDD. We utilize conv-transformer structure to encode input speech in a streaming manner. A coupled cross-attention (CoCA) mechanism is proposed to integrate frame-level acoustic features with encoded reference linguistic features. CoCA also enables our model to perform mispronunciation classification with whole utterances. The proposed model allows system fusion between the streaming output and mispronunciation classification output for further performance enhancement. We evaluate CoCA-MDD on publicly available corpora. CoCA-MDD achieves F1 scores of 57.03% and 60.78% for streaming and fusion mode

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyzheng2021coca

Related papers

• A Full Text-dependent End To End Mispronunciation Detection And Diagnosis With Easy Data Augmentation Techniques (2021)0.00
• Multi-view Multi-task Representation Learning For Mispronunciation Detection (2023)0.00
• Improving Mispronunciation Detection With Wav2vec2-based Momentum Pseudo-labeling For Accentedness And Intelligibility Assessment (2022)7.16
• Adaptive Frequency Cepstral Coefficients For Word Mispronunciation Detection (2016)5.84
• Alignment Knowledge Distillation For Online Streaming Attention-based Speech Recognition (2021)7.16
• Improving End-to-end Modeling For Mispronunciation Detection With Effective Augmentation Mechanisms (2021)0.00
• Unified Streaming And Non-streaming Two-pass End-to-end Model For Speech Recognition (2020)0.00
• Multiview Canonical Correlation Analysis For Automatic Pathological Speech Detection (2024)2.26