Interpretable Style Transfer For Text-to-speech With Controlvae And Diffusion Bridge

Abstract

With the demand for autonomous control and personalized speech generation, the style control and transfer in Text-to-Speech (TTS) is becoming more and more important. In this paper, we propose a new TTS system that can perform style transfer with interpretability and high fidelity. Firstly, we design a TTS system that combines variational autoencoder (VAE) and diffusion refiner to get refined mel-spectrograms. Specifically, a two-stage and a one-stage system are designed respectively, to improve the audio quality and the performance of style transfer. Secondly, a diffusion bridge of quantized VAE is designed to efficiently learn complex discrete style representations and improve the performance of style transfer. To have a better ability of style transfer, we introduce ControlVAE to improve the reconstruction quality and have good interpretability simultaneously. Experiments on LibriTTS dataset demonstrate that our method is more effective than baseline models.

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyguan2023interpretable

Related papers

• Learning Latent Representations For Style Control And Transfer In End-to-end Speech Synthesis (2018)0.00
• Fine-grained Style Control In Transformer-based Text-to-speech Synthesis (2021)11.19
• Text-driven Emotional Style Control And Cross-speaker Style Transfer In Neural TTS (2022)7.81
• Styletts 2: Towards Human-level Text-to-speech Through Style Diffusion And Adversarial Training With Large Speech Language Models (2023)8.09
• Promptstyle: Controllable Style Transfer For Text-to-speech With Natural Language Descriptions (2023)10.85
• Style-label-free: Cross-speaker Style Transfer By Quantized VAE And Speaker-wise Normalization In Speech Synthesis (2022)4.52
• Exploring Synthetic Data For Cross-speaker Style Transfer In Style Representation Based TTS (2024)0.00
• Improving Performance Of Seen And Unseen Speech Style Transfer In End-to-end Neural TTS (2021)6.34