VQMIVC: Vector Quantization And Mutual Information-based Unsupervised Speech Representation Disentanglement For One-shot Voice Conversion

Abstract

One-shot voice conversion (VC), which performs conversion across arbitrary speakers with only a single target-speaker utterance for reference, can be effectively achieved by speech representation disentanglement. Existing work generally ignores the correlation between different speech representations during training, which causes leakage of content information into the speaker representation and thus degrades VC performance. To alleviate this issue, we employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training, to achieve proper disentanglement of content, speaker and pitch representations, by reducing their inter-dependencies in an unsupervised manner. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations for retaining source linguistic content and intonation variations, while capturing target speaker characteristics. In doing so, the propos

Stats

• citations105
• S2 citations—
• github stars361
• HF likes0
• heat score20.31
• arxiv keywang2021vqmivc

Related papers

• Speech Representation Disentanglement With Adversarial Mutual Information Learning For One-shot Voice Conversion (2022)11.08
• Robust Disentangled Variational Speech Representation Learning For Zero-shot Voice Conversion (2022)10.97
• AVQVC: One-shot Voice Conversion By Vector Quantization With Applying Contrastive Learning (2022)12.40
• QR-VC: Leveraging Quantization Residuals For Linear Disentanglement In Zero-shot Voice Conversion (2024)0.00
• VQVC+: One-shot Voice Conversion By Vector Quantization And U-net Architecture (2020)13.34
• MAIN-VC: Lightweight Speech Representation Disentanglement For One-shot Voice Conversion (2024)3.58
• One-shot Voice Conversion By Separating Speaker And Content Representations With Instance Normalization (2019)0.00
• Zero-shot Voice Conversion Via Self-supervised Prosody Representation Learning (2021)6.34