Sample Efficient Adaptive Text-to-speech

Abstract

We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keychen2018sample

Related papers

• Meta-tts: Meta-learning For Few-shot Speaker Adaptive Text-to-speech (2021)12.74
• Speaker-adaptive Neural Vocoders For Parametric Speech Synthesis Systems (2018)2.26
• Adapter-based Extension Of Multi-speaker Text-to-speech Model For New Speakers (2022)6.77
• Adapting TTS Models For New Speakers Using Transfer Learning (2021)0.00
• Rapid Speaker Adaptation In Low Resource Text To Speech Systems Using Synthetic Data And Transfer Learning (2023)0.00
• Transfer Learning From Speaker Verification To Multispeaker Text-to-speech Synthesis (2018)0.00
• Nanovoice: Efficient Speaker-adaptive Text-to-speech For Multiple Speakers (2024)0.00
• A Unified Speaker Adaptation Method For Speech Synthesis Using Transcribed And Untranscribed Speech With Backpropagation (2019)0.00