Voicetailor: Lightweight Plug-in Adapter For Diffusion-based Personalized Text-to-speech

Abstract

We propose VoiceTailor, a parameter-efficient speaker-adaptive text-to-speech (TTS) system, by equipping a pre-trained diffusion-based TTS model with a personalized adapter. VoiceTailor identifies pivotal modules that benefit from the adapter based on a weight change ratio analysis. We utilize Low-Rank Adaptation (LoRA) as a parameter-efficient adaptation method and incorporate the adapter into pivotal modules of the pre-trained diffusion decoder. To achieve powerful adaptation performance with few parameters, we explore various guidance techniques for speaker adaptation and investigate the best strategies to strengthen speaker information. VoiceTailor demonstrates comparable speaker adaptation performance to existing adaptive TTS models by fine-tuning only 0.25% of the total parameters. VoiceTailor shows strong robustness when adapting to a wide range of real-world speakers, as shown in the demo.

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keykim2024voicetailor

Related papers

• Adapter-based Extension Of Multi-speaker Text-to-speech Model For New Speakers (2022)6.77
• ADAPTERMIX: Exploring The Efficacy Of Mixture Of Adapters For Low-resource TTS Adaptation (2023)6.34
• Voiceguider: Enhancing Out-of-domain Performance In Parameter-efficient Speaker-adaptive Text-to-speech Via Autoguidance (2024)0.00
• Nanovoice: Efficient Speaker-adaptive Text-to-speech For Multiple Speakers (2024)0.00
• Guided-tts 2: A Diffusion Model For High-quality Adaptive Text-to-speech With Untranscribed Data (2022)0.00
• Leveraging Parameter-efficient Transfer Learning For Multi-lingual Text-to-speech Adaptation (2024)0.00
• Grad-stylespeech: Any-speaker Adaptive Text-to-speech Synthesis With Diffusion Models (2022)0.00
• TDASS: Target Domain Adaptation Speech Synthesis Framework For Multi-speaker Low-resource TTS (2022)0.00