Fine-grained Robust Prosody Transfer For Single-speaker Neural Text-to-speech

Abstract

We present a neural text-to-speech system for fine-grained prosody transfer from one speaker to another. Conventional approaches for end-to-end prosody transfer typically use either fixed-dimensional or variable-length prosody embedding via a secondary attention to encode the reference signal. However, when trained on a single-speaker dataset, the conventional prosody transfer systems are not robust enough to speaker variability, especially in the case of a reference signal coming from an unseen speaker. Therefore, we propose decoupling of the reference signal alignment from the overall system. For this purpose, we pre-compute phoneme-level time stamps and use them to aggregate prosodic features per phoneme, injecting them into a sequence-to-sequence text-to-speech system. We incorporate a variational auto-encoder to further enhance the latent representation of prosody embeddings. We show that our proposed approach is significantly more stable and achieves reliable prosody transplantat

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyklimkov2019fine

Related papers

• Robust And Fine-grained Prosody Control Of End-to-end Speech Synthesis (2018)14.31
• Copycat: Many-to-many Fine-grained Prosody Transfer For Neural Text-to-speech (2020)11.76
• Improving Multi-speaker TTS Prosody Variance With A Residual Encoder And Normalizing Flows (2021)0.00
• Towards Fine-grained Prosody Control For Voice Conversion (2019)6.34
• Adversarial Learning Of Intermediate Acoustic Feature For End-to-end Lightweight Text-to-speech (2022)0.00
• Prosody Transfer In Neural Text To Speech Using Global Pitch And Loudness Features (2019)0.00
• Cross-speaker Style Transfer With Prosody Bottleneck In Neural Speech Synthesis (2021)10.21
• Towards End-to-end Prosody Transfer For Expressive Speech Synthesis With Tacotron (2018)0.00