Zero-shot Sing Voice Conversion: Built Upon Clustering-based Phoneme Representations

Abstract

This study presents an innovative Zero-Shot any-to-any Singing Voice Conversion (SVC) method, leveraging a novel clustering-based phoneme representation to effectively separate content, timbre, and singing style. This approach enables precise voice characteristic manipulation. We discovered that datasets with fewer recordings per artist are more susceptible to timbre leakage. Extensive testing on over 10,000 hours of singing and user feedback revealed our model significantly improves sound quality and timbre accuracy, aligning with our objectives and advancing voice conversion technology. Furthermore, this research advances zero-shot SVC and sets the stage for future work on discrete speech representation, emphasizing the preservation of rhyme.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhou2024zero

Related papers

• Everyone-can-sing: Zero-shot Singing Voice Synthesis And Conversion With Speech Reference (2025)0.00
• Samoye: Zero-shot Singing Voice Conversion Model Based On Feature Disentanglement And Enhancement (2024)3.50
• LDM-SVC: Latent Diffusion Model Based Zero-shot Any-to-any Singing Voice Conversion With Singer Guidance (2024)5.84
• Neural Concatenative Singing Voice Conversion: Rethinking Concatenation-based Approach For One-shot Singing Voice Conversion (2023)7.50
• Zero-shot Voice Conversion Via Self-supervised Prosody Representation Learning (2021)6.34
• Robust One-shot Singing Voice Conversion (2022)0.00
• Leveraging Diverse Semantic-based Audio Pretrained Models For Singing Voice Conversion (2023)0.00
• Ppg-based Singing Voice Conversion With Adversarial Representation Learning (2020)9.76