Affectecho: Speaker Independent And Language-agnostic Emotion And Affect Transfer For Speech Synthesis

Abstract

Affect is an emotional characteristic encompassing valence, arousal, and intensity, and is a crucial attribute for enabling authentic conversations. While existing text-to-speech (TTS) and speech-to-speech systems rely on strength embedding vectors and global style tokens to capture emotions, these models represent emotions as a component of style or represent them in discrete categories. We propose AffectEcho, an emotion translation model, that uses a Vector Quantized codebook to model emotions within a quantized space featuring five levels of affect intensity to capture complex nuances and subtle differences in the same emotion. The quantized emotional embeddings are implicitly derived from spoken speech samples, eliminating the need for one-hot vectors or explicit strength embeddings. Experimental results demonstrate the effectiveness of our approach in controlling the emotions of generated speech while preserving identity, style, and emotional cadence unique to each speaker. We sho

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyviswanath2023affectecho

Related papers

• Cross-speaker Emotion Disentangling And Transfer For End-to-end Speech Synthesis (2021)12.61
• Iemotts: Toward Robust Cross-speaker Emotion Transfer And Control For Speech Synthesis Based On Disentanglement Between Prosody And Timbre (2022)0.00
• Msemotts: Multi-scale Emotion Transfer, Prediction, And Control For Emotional Speech Synthesis (2022)13.97
• Fine-grained Emotion Strength Transfer, Control And Prediction For Emotional Speech Synthesis (2020)12.25
• Emosphere-tts: Emotional Style And Intensity Modeling Via Spherical Emotion Vector For Controllable Emotional Text-to-speech (2024)10.35
• Mixed-evc: Mixed Emotion Synthesis And Control In Voice Conversion (2022)4.52
• Emomix: Emotion Mixing Via Diffusion Models For Emotional Speech Synthesis (2023)0.00
• METTS: Multilingual Emotional Text-to-speech By Cross-speaker And Cross-lingual Emotion Transfer (2023)0.00