Towards Expressive Zero-shot Speech Synthesis With Hierarchical Prosody Modeling

Abstract

Recent research in zero-shot speech synthesis has made significant progress in speaker similarity. However, current efforts focus on timbre generalization rather than prosody modeling, which results in limited naturalness and expressiveness. To address this, we introduce a novel speech synthesis model trained on large-scale datasets, including both timbre and hierarchical prosody modeling. As timbre is a global attribute closely linked to expressiveness, we adopt a global vector to model speaker timbre while guiding prosody modeling. Besides, given that prosody contains both global consistency and local variations, we introduce a diffusion model as the pitch predictor and employ a prosody adaptor to model prosody hierarchically, further enhancing the prosody quality of the synthesized speech. Experimental results show that our model not only maintains comparable timbre quality to the baseline but also exhibits better naturalness and expressiveness.

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyjiang2024towards

Related papers

• Hierarchical Prosody Modeling For Non-autoregressive Speech Synthesis (2020)10.07
• Hierspeech++: Bridging The Gap Between Semantic And Acoustic Representation Of Speech By Hierarchical Variational Inference For Zero-shot Speech Synthesis (2023)6.19
• Towards Expressive Speaking Style Modelling With Hierarchical Context Information For Mandarin Speech Synthesis (2022)6.34
• Hierarchical Multi-grained Generative Model For Expressive Speech Synthesis (2020)8.60
• Hierarchical Prosody Modeling And Control In Non-autoregressive Parallel Neural TTS (2021)8.35
• Fully-hierarchical Fine-grained Prosody Modeling For Interpretable Speech Synthesis (2020)13.28
• Improving Prosody For Cross-speaker Style Transfer By Semi-supervised Style Extractor And Hierarchical Modeling In Speech Synthesis (2023)7.50
• HAM-TTS: Hierarchical Acoustic Modeling For Token-based Zero-shot Text-to-speech With Model And Data Scaling (2024)0.00