Controllable Emotion Transfer For End-to-end Speech Synthesis

Abstract

Emotion embedding space learned from references is a straightforward approach for emotion transfer in encoder-decoder structured emotional text to speech (TTS) systems. However, the transferred emotion in the synthetic speech is not accurate and expressive enough with emotion category confusions. Moreover, it is hard to select an appropriate reference to deliver desired emotion strength. To solve these problems, we propose a novel approach based on Tacotron. First, we plug two emotion classifiers -- one after the reference encoder, one after the decoder output -- to enhance the emotion-discriminative ability of the emotion embedding and the predicted mel-spectrum. Second, we adopt style loss to measure the difference between the generated and reference mel-spectrum. The emotion strength in the synthetic speech can be controlled by adjusting the value of the emotion embedding as the emotion embedding can be viewed as the feature map of the mel-spectrum. Experiments on emotion transfer a

Stats

• citations54
• S2 citations—
• github stars0
• HF likes0
• heat score13.05
• arxiv keyli2020controllable

Related papers

• Cross-speaker Emotion Disentangling And Transfer For End-to-end Speech Synthesis (2021)12.61
• Towards End-to-end Prosody Transfer For Expressive Speech Synthesis With Tacotron (2018)0.00
• Iemotts: Toward Robust Cross-speaker Emotion Transfer And Control For Speech Synthesis Based On Disentanglement Between Prosody And Timbre (2022)0.00
• Generative Emotional AI For Speech Emotion Recognition: The Case For Synthetic Emotional Speech Augmentation (2023)11.19
• Fine-grained Emotion Strength Transfer, Control And Prediction For Emotional Speech Synthesis (2020)12.25
• Text-driven Emotional Style Control And Cross-speaker Style Transfer In Neural TTS (2022)7.81
• Emotional Speech Synthesis With Rich And Granularized Control (2019)13.39
• RSET: Remapping-based Sorting Method For Emotion Transfer Speech Synthesis (2024)0.00