Towards General-purpose Text-instruction-guided Voice Conversion

Abstract

This paper introduces a novel voice conversion (VC) model, guided by text instructions such as "articulate slowly with a deep tone" or "speak in a cheerful boyish voice". Unlike traditional methods that rely on reference utterances to determine the attributes of the converted speech, our model adds versatility and specificity to voice conversion. The proposed VC model is a neural codec language model which processes a sequence of discrete codes, resulting in the code sequence of converted speech. It utilizes text instructions as style prompts to modify the prosody and emotional information of the given speech. In contrast to previous approaches, which often rely on employing separate encoders like prosody and content encoders to handle different aspects of the source speech, our model handles various information of speech in an end-to-end manner. Experiments have demonstrated the impressive capabilities of our model in comprehending instructions and delivering reasonable results.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykuan2023towards

Related papers

• Fastvc: Fast Voice Conversion With Non-parallel Data (2020)5.24
• PMVC: Data Augmentation-based Prosody Modeling For Expressive Voice Conversion (2023)9.23
• Controlvc: Zero-shot Voice Conversion With Time-varying Controls On Pitch And Speed (2022)6.77
• Beyond Voice Identity Conversion: Manipulating Voice Attributes By Adversarial Learning Of Structured Disentangled Representations (2021)0.00
• AC-VC: Non-parallel Low Latency Phonetic Posteriorgrams Based Voice Conversion (2021)7.50
• Generative Adversarial Network Based Voice Conversion: Techniques, Challenges, And Recent Advancements (2025)0.00
• Assem-vc: Realistic Voice Conversion By Assembling Modern Speech Synthesis Techniques (2021)11.64
• Emotional Voice Conversion Using Multitask Learning With Text-to-speech (2019)0.00