Colm-dsr: Leveraging Neural Codec Language Modeling For Multi-modal Dysarthric Speech Reconstruction

Abstract

Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech. It still suffers from low speaker similarity and poor prosody naturalness. In this paper, we propose a multi-modal DSR model by leveraging neural codec language modeling to improve the reconstruction results, especially for the speaker similarity and prosody naturalness. Our proposed model consists of: (i) a multi-modal content encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual inputs; (ii) a speaker codec encoder to extract and normalize the speaker-aware codecs from the dysarthric speech, in order to provide original timbre and normal prosody; (iii) a codec language model based speech decoder to reconstruct the speech based on the extracted phoneme embeddings and normalized codecs. Evaluations on the commonly used UASpeech corpus show that our proposed model can achieve significant improvements in terms of speaker similarity and prosody naturalness.

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keychen2024colm

Related papers

• UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization (2024)0.00
• Speaker Identity Preservation In Dysarthric Speech Reconstruction By Adversarial Speaker Adaptation (2022)0.00
• Learning Explicit Prosody Models And Deep Speaker Embeddings For Atypical Voice Conversion (2020)7.16
• Investigating Neural Audio Codecs For Speech Language Model-based Speech Generation (2024)2.26
• Discrete Multimodal Transformers With A Pretrained Large Language Model For Mixed-supervision Speech Processing (2024)0.00
• Enhancing Dysarthric Speech Recognition For Unseen Speakers Via Prototype-based Adaptation (2024)9.45
• Analyzing And Mitigating Inconsistency In Discrete Audio Tokens For Neural Codec Language Models (2024)5.84
• Language-codec: Bridging Discrete Codec Representations And Speech Language Models (2024)4.64