Predicting Phoneme-level Prosody Latents Using AR And Flow-based Prior Networks For Expressive Speech Synthesis

Abstract

A large part of the expressive speech synthesis literature focuses on learning prosodic representations of the speech signal which are then modeled by a prior distribution during inference. In this paper, we compare different prior architectures at the task of predicting phoneme level prosodic representations extracted with an unsupervised FVAE model. We use both subjective and objective metrics to show that normalizing flow based prior networks can result in more expressive speech at the cost of a slight drop in quality. Furthermore, we show that the synthesized speech has higher variability, for a given text, due to the nature of normalizing flows. We also propose a Dynamical VAE model, that can generate higher quality speech although with decreased expressiveness and variability compared to the flow based models.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyklapsas2022predicting

Related papers

• Generating Diverse And Natural Text-to-speech Samples Using A Quantized Fine-grained VAE And Auto-regressive Prosody Prior (2020)12.54
• Comparing Normalizing Flows And Diffusion Models For Prosody And Acoustic Modelling In Text-to-speech (2023)0.00
• Using Vaes And Normalizing Flows For One-shot Text-to-speech Synthesis Of Expressive Speech (2019)9.92
• Glow-wavegan: Learning Speech Representations From Gan-based Variational Auto-encoder For High Fidelity Flow-based Speech Synthesis (2021)8.35
• Improving Multi-speaker TTS Prosody Variance With A Residual Encoder And Normalizing Flows (2021)0.00
• Learning Utterance-level Representations Through Token-level Acoustic Latents Prediction For Expressive Speech Synthesis (2022)0.00
• Generative Modeling For Low Dimensional Speech Attributes With Neural Spline Flows (2022)0.00
• Generative Pre-training For Speech With Flow Matching (2023)0.00