Instructtts: Modelling Expressive TTS In Discrete Latent Space With Natural Language Style Prompt

Abstract

Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, e.g., "Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyang2023instructtts

Related papers

• Expressive TTS Driven By Natural Language Prompts Using Few Human Annotations (2023)0.00
• Expressive Text-to-speech Using Style Tag (2021)10.85
• Styletts: A Style-based Generative Model For Natural And Diverse Text-to-speech Synthesis (2022)10.97
• MM-TTS: Multi-modal Prompt Based Style Transfer For Expressive Text-to-speech Synthesis (2023)8.60
• Textrolspeech: A Text Style Control Speech Corpus With Codec Language Text-to-speech Models (2023)9.59
• Promptstyle: Controllable Style Transfer For Text-to-speech With Natural Language Descriptions (2023)10.85
• DEX-TTS: Diffusion-based Expressive Text-to-speech With Style Modeling On Time Variability (2024)0.00
• Self-supervised Context-aware Style Representation For Expressive Speech Synthesis (2022)6.34