Robust And Fine-grained Prosody Control Of End-to-end Speech Synthesis

Abstract

We propose prosody embeddings for emotional and expressive speech synthesis networks. The proposed methods introduce temporal structures in the embedding networks, thus enabling fine-grained control of the speaking style of the synthesized speech. The temporal structures can be designed either on the speech side or the text side, leading to different control resolutions in time. The prosody embedding networks are plugged into end-to-end speech synthesis networks and trained without any other supervision except for the target speech for synthesizing. It is demonstrated that the prosody embedding networks learned to extract prosodic features. By adjusting the learned prosody features, we could change the pitch and amplitude of the synthesized speech both at the frame level and the phoneme level. We also introduce the temporal normalization of prosody embeddings, which shows better robustness against speaker perturbations during prosody transfer tasks.

Stats

• citations80
• S2 citations—
• github stars0
• HF likes0
• heat score14.31
• arxiv keylee2018robust

Related papers

• Fine-grained Robust Prosody Transfer For Single-speaker Neural Text-to-speech (2019)0.00
• Towards Fine-grained Prosody Control For Voice Conversion (2019)6.34
• PROEMO: Prompt-driven Text-to-speech Synthesis Based On Emotion And Intensity Control (2025)0.00
• Emotional Speech Synthesis With Rich And Granularized Control (2019)13.39
• Fine-grained Emotion Strength Transfer, Control And Prediction For Emotional Speech Synthesis (2020)12.25
• Iemotts: Toward Robust Cross-speaker Emotion Transfer And Control For Speech Synthesis Based On Disentanglement Between Prosody And Timbre (2022)0.00
• Dynamic Prosody Generation For Speech Synthesis Using Linguistics-driven Acoustic Embedding Selection (2019)7.81
• Controllable Speech Synthesis By Learning Discrete Phoneme-level Prosodic Representations (2022)6.34