Structured Speaker-deficiency Adaptation Of Foundation Models For Dysarthric And Elderly Speech Recognition

Abstract

Data-intensive fine-tuning of speech foundation models (SFMs) to scarce and diverse dysarthric and elderly speech leads to data bias and poor generalization to unseen speakers. This paper proposes novel structured speaker-deficiency adaptation approaches for SSL pre-trained SFMs on such data. Speaker and speech deficiency invariant SFMs were constructed in their supervised adaptive fine-tuning stage to reduce undue bias to training data speakers, and serves as a more neutral and robust starting point for test time unsupervised adaptation. Speech variability attributed to speaker identity and speech impairment severity, or aging induced neurocognitive decline, are modelled using separate adapters that can be combined together to model any seen or unseen speaker. Experiments on the UASpeech dysarthric and DementiaBank Pitt elderly speech corpora suggest structured speaker-deficiency adaptation of HuBERT and Wav2vec2-conformer models consistently outperforms baseline SFMs using either: a)

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhu2024structured

Related papers

• On-the-fly Feature Based Rapid Speaker Adaptation For Dysarthric And Elderly Speech Recognition (2022)6.34
• Homogeneous Speaker Features For On-the-fly Dysarthric And Elderly Speaker Adaptation (2024)0.00
• Speaker Adaptation Using Spectro-temporal Deep Features For Dysarthric And Elderly Speech Recognition (2022)12.02
• Hyper-parameter Adaptation Of Conformer ASR Systems For Elderly And Dysarthric Speech Recognition (2023)0.00
• Enhancing Dysarthric Speech Recognition For Unseen Speakers Via Prototype-based Adaptation (2024)9.45
• Resource-efficient Adaptation Of Speech Foundation Models For Multi-speaker ASR (2024)3.58
• Personalized Adversarial Data Augmentation For Dysarthric And Elderly Speech Recognition (2022)11.49
• Unsupervised Model-based Speaker Adaptation Of End-to-end Lattice-free MMI Model For Speech Recognition (2022)2.26