Norespeech: Knowledge Distillation Based Conditional Diffusion Model For Noise-robust Expressive TTS

Abstract

Expressive text-to-speech (TTS) can synthesize a new speaking style by imiating prosody and timbre from a reference audio, which faces the following challenges: (1) The highly dynamic prosody information in the reference audio is difficult to extract, especially, when the reference audio contains background noise. (2) The TTS systems should have good generalization for unseen speaking styles. In this paper, we present a \textbf\{no\}ise-\textbf\{r\}obust \textbf\{e\}xpressive TTS model (NoreSpeech), which can robustly transfer speaking style in a noisy reference utterance to synthesized speech. Specifically, our NoreSpeech includes several components: (1) a novel DiffStyle module, which leverages powerful probabilistic denoising diffusion models to learn noise-agnostic speaking style features from a teacher model by knowledge distillation; (2) a VQ-VAE block, which maps the style features into a controllable quantized latent space for improving the generalization of style transfer; and

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyang2022norespeech

Related papers

• Styletts-zs: Efficient High-quality Zero-shot Text-to-speech Synthesis With Distilled Time-varying Style Diffusion (2024)3.58
• DEX-TTS: Diffusion-based Expressive Text-to-speech With Style Modeling On Time Variability (2024)0.00
• Diffstyletts: Diffusion-based Hierarchical Prosody Modeling For Text-to-speech With Diverse And Controllable Styles (2024)0.00
• Diffprosody: Diffusion-based Latent Prosody Generation For Expressive Speech Synthesis With Prosody Conditional Adversarial Training (2023)10.07
• Grad-stylespeech: Any-speaker Adaptive Text-to-speech Synthesis With Diffusion Models (2022)0.00
• High-fidelity Speech Synthesis With Minimal Supervision: All Using Diffusion Models (2023)5.24
• Style Description Based Text-to-speech With Conditional Prosodic Layer Normalization Based Diffusion GAN (2023)0.00
• STYLER: Style Factor Modeling With Rapidity And Robustness Via Speech Decomposition For Expressive And Controllable Neural Text To Speech (2021)9.23