Diffprosody: Diffusion-based Latent Prosody Generation For Expressive Speech Synthesis With Prosody Conditional Adversarial Training

Abstract

Expressive text-to-speech systems have undergone significant advancements owing to prosody modeling, but conventional methods can still be improved. Traditional approaches have relied on the autoregressive method to predict the quantized prosody vector; however, it suffers from the issues of long-term dependency and slow inference. This study proposes a novel approach called DiffProsody in which expressive speech is synthesized using a diffusion-based latent prosody generator and prosody conditional adversarial training. Our findings confirm the effectiveness of our prosody generator in generating a prosody vector. Furthermore, our prosody conditional discriminator significantly improves the quality of the generated speech by accurately emulating prosody. We use denoising diffusion generative adversarial networks to improve the prosody generation speed. Consequently, DiffProsody is capable of generating prosody 16 times faster than the conventional diffusion model. The superior perform

Stats

• citations21
• S2 citations—
• github stars0
• HF likes0
• heat score10.07
• arxiv keyoh2023diffprosody

Related papers

• Prodiff: Progressive Fast Diffusion Model For High-quality Text-to-speech (2022)0.00
• Diverse And Expressive Speech Prosody Prediction With Denoising Diffusion Probabilistic Model (2023)4.52
• Fastdiff: A Fast Conditional Diffusion Model For High-quality Speech Synthesis (2022)14.35
• Diffstyletts: Diffusion-based Hierarchical Prosody Modeling For Text-to-speech With Diverse And Controllable Styles (2024)0.00
• Diffar: Denoising Diffusion Autoregressive Model For Raw Speech Waveform Generation (2023)0.00
• Diffgan-tts: High-fidelity And Efficient Text-to-speech With Denoising Diffusion Gans (2022)0.00
• Style Description Based Text-to-speech With Conditional Prosodic Layer Normalization Based Diffusion GAN (2023)0.00
• Diffmotion: Speech-driven Gesture Synthesis Using Denoising Diffusion Model (2023)9.59