SLMGAN: Exploiting Speech Language Model Representations For Unsupervised Zero-shot Voice Conversion In Gans
2023 · Yinghao Aaron Li, Cong Han, Nima Mesgarani
Abstract
In recent years, large-scale pre-trained speech language models (SLMs) have demonstrated remarkable advancements in various generative speech modeling applications, such as text-to-speech synthesis, voice conversion, and speech enhancement. These applications typically involve mapping text or speech inputs to pre-trained SLM representations, from which target speech is decoded. This paper introduces a new approach, SLMGAN, to leverage SLM representations for discriminative tasks within the generative adversarial network (GAN) framework, specifically for voice conversion. Building upon StarGANv2-VC, we add our novel SLM-based WavLM discriminators on top of the mel-based discriminators along with our newly designed SLM feature matching loss function, resulting in an unsupervised zero-shot voice conversion system that does not require text labels during training. Subjective evaluation results show that SLMGAN outperforms existing state-of-the-art zero-shot voice conversion models in terms
Authors
(none)
Tags
Stats
Related papers
- Stargan-zsvc: Towards Zero-shot Voice Conversion In Low-resource Contexts (2021)3.58
- LM-VC: Zero-shot Voice Conversion Via Speech Generation Based On Language Models (2023)0.00
- GAZEV: Gan-based Zero-shot Voice Conversion Over Non-parallel Speech Corpus (2020)8.60
- Subband-based Generative Adversarial Network For Non-parallel Many-to-many Voice Conversion (2022)0.00
- Starganv2-vc: A Diverse, Unsupervised, Non-parallel Framework For Natural-sounding Voice Conversion (2021)13.70
- Stargan-vc: Non-parallel Many-to-many Voice Conversion With Star Generative Adversarial Networks (2018)18.09
- Collective Learning Mechanism Based Optimal Transport Generative Adversarial Network For Non-parallel Voice Conversion (2025)0.00
- Harnessing Gans For Zero-shot Learning Of New Classes In Visual Speech Recognition (2019)8.09