Rethinking Speaker Embeddings For Speech Generation: Sub-center Modeling For Capturing Intra-speaker Diversity

Abstract

Modeling the rich prosodic variations inherent in human speech is essential for generating natural-sounding speech. While speaker embeddings are commonly used as conditioning inputs in personalized speech generation, they are typically optimized for speaker recognition, which encourages the loss of intra-speaker variation. This strategy makes them suboptimal for speech generation in terms of modeling the rich variations at the output speech distribution. In this work, we propose a novel speaker embedding network that employs multiple sub-centers per speaker class during training, instead of a single center as in conventional approaches. This sub-center modeling allows the embedding to capture a broader range of speaker-specific variations while maintaining speaker classification performance. We demonstrate the effectiveness of the proposed embeddings on a voice conversion task, showing improved naturalness and prosodic expressiveness in the synthesized speech.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyulgen2024rethinking

Related papers

• Controllable Generation Of Artificial Speaker Embeddings Through Discovery Of Principal Directions (2023)0.00
• An Analysis On The Effects Of Speaker Embedding Choice In Non Auto-regressive TTS (2023)0.00
• Rethinking Session Variability: Leveraging Session Embeddings For Session Robustness In Speaker Verification (2023)5.24
• Robust And Fine-grained Prosody Control Of End-to-end Speech Synthesis (2018)14.31
• Using Multiple Reference Audios And Style Embedding Constraints For Speech Synthesis (2021)5.24
• Unispeaker: A Unified Approach For Multimodality-driven Speaker Generation (2025)2.26
• Learning Explicit Prosody Models And Deep Speaker Embeddings For Atypical Voice Conversion (2020)7.16
• Diffv2s: Diffusion-based Video-to-speech Synthesis With Vision-guided Speaker Embedding (2023)8.82