Prompttts++: Controlling Speaker Identity In Prompt-based Text-to-speech Using Natural Language Descriptions

Abstract

We propose PromptTTS++, a prompt-based text-to-speech (TTS) synthesis system that allows control over speaker identity using natural language descriptions. To control speaker identity within the prompt-based TTS framework, we introduce the concept of speaker prompt, which describes voice characteristics (e.g., gender-neutral, young, old, and muffled) designed to be approximately independent of speaking style. Since there is no large-scale dataset containing speaker prompts, we first construct a dataset based on the LibriTTS-R corpus with manually annotated speaker prompts. We then employ a diffusion-based acoustic model with mixture density networks to model diverse speaker factors in the training data. Unlike previous studies that rely on style prompts describing only a limited aspect of speaker individuality, such as pitch, speaking speed, and energy, our method utilizes an additional speaker prompt to effectively learn the mapping from natural language descriptions to the acoustic f

Stats

• citations16
• S2 citations—
• github stars0
• HF likes0
• heat score9.23
• arxiv keyshimizu2023prompttts

Related papers

• Generating Speakers By Prompting Listener Impressions For Pre-trained Multi-speaker Text-to-speech Systems (2024)3.58
• Prompt-singer: Controllable Singing-voice-synthesis With Natural Language Prompt (2024)6.77
• Libritts-p: A Corpus With Speaking Style And Speaker Identity Prompts For Text-to-speech And Style Captioning (2024)11.91
• Promptstyle: Controllable Style Transfer For Text-to-speech With Natural Language Descriptions (2023)10.85
• Instructtts: Modelling Expressive TTS In Discrete Latent Space With Natural Language Style Prompt (2023)0.00
• Expressive Prompting: Improving Emotion Intensity And Speaker Consistency In Zero-shot TTS (2024)0.00
• Building Speech Corpus With Diverse Voice Characteristics For Its Prompt-based Representation (2024)0.00
• Natural Language Guidance Of High-fidelity Text-to-speech With Synthetic Annotations (2024)0.00