Zero-shot Voice Conversion Via Self-supervised Prosody Representation Learning

Abstract

Voice Conversion (VC) for unseen speakers, also known as zero-shot VC, is an attractive research topic as it enables a range of applications like voice customizing, animation production, and others. Recent work in this area made progress with disentanglement methods that separate utterance content and speaker characteristics from speech audio recordings. However, many of these methods are subject to the leakage of prosody (e.g., pitch, volume), causing the speaker voice in the synthesized speech to be different from the desired target speakers. To prevent this issue, we propose a novel self-supervised approach that effectively learns disentangled pitch and volume representations that can represent the prosody styles of different speakers. We then use the learned prosodic representations as conditional information to train and enhance our VC model for zero-shot conversion. In our experiments, we show that our prosody representations are disentangled and rich in prosody information. More

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keywang2021zero

Related papers

• Robust Disentangled Variational Speech Representation Learning For Zero-shot Voice Conversion (2022)10.97
• ACE-VC: Adaptive And Controllable Voice Conversion Using Explicitly Disentangled Self-supervised Speech Representations (2023)0.00
• Training Robust Zero-shot Voice Conversion Models With Self-supervised Features (2021)7.16
• Disentangling The Prosody And Semantic Information With Pre-trained Model For In-context Learning Based Zero-shot Voice Conversion (2024)4.52
• Voicy: Zero-shot Non-parallel Voice Conversion In Noisy Reverberant Environments (2021)5.24
• QR-VC: Leveraging Quantization Residuals For Linear Disentanglement In Zero-shot Voice Conversion (2024)0.00
• Speech Representation Disentanglement With Adversarial Mutual Information Learning For One-shot Voice Conversion (2022)11.08
• ZSVC: Zero-shot Style Voice Conversion With Disentangled Latent Diffusion Models And Adversarial Training (2025)0.00