UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization

Abstract

Dysarthric speech reconstruction (DSR) systems aim to automatically convert dysarthric speech into normal-sounding speech. The technology eases communication with speakers affected by the neuromotor disorder and enhances their social inclusion. NED-based (Neural Encoder-Decoder) systems have significantly improved the intelligibility of the reconstructed speech as compared with GAN-based (Generative Adversarial Network) approaches, but the approach is still limited by training inefficiency caused by the cascaded pipeline and auxiliary tasks of the content encoder, which may in turn affect the quality of reconstruction. Inspired by self-supervised speech representation learning and discrete speech units, we propose a Unit-DSR system, which harnesses the powerful domain-adaptation capacity of HuBERT for training efficiency improvement and utilizes speech units to constrain the dysarthric content restoration in a discrete linguistic space. Compared with NED approaches, the Unit-DSR system

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2024unit

Related papers

• Colm-dsr: Leveraging Neural Codec Language Modeling For Multi-modal Dysarthric Speech Reconstruction (2024)3.58
• Enhancing Dysarthric Speech Recognition For Unseen Speakers Via Prototype-based Adaptation (2024)9.45
• Speaker Identity Preservation In Dysarthric Speech Reconstruction By Adversarial Speaker Adaptation (2022)0.00
• Domain Adversarial Neural Networks For Dysarthric Speech Recognition (2020)7.50
• Exploring Speech Recognition, Translation, And Understanding With Discrete Speech Units: A Comparative Study (2023)0.00
• Unity: Two-pass Direct Speech-to-speech Translation With Discrete Units (2022)9.59
• DENT-DDSP: Data-efficient Noisy Speech Generator Using Differentiable Digital Signal Processors For Explicit Distortion Modelling And Noise-robust Speech Recognition (2022)0.00
• Discreteslu: A Large Language Model With Self-supervised Discrete Speech Units For Spoken Language Understanding (2024)5.84