Semi-supervised Learning For Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation

Abstract

Recently, end-to-end multi-speaker text-to-speech (TTS) systems gain success in the situation where a lot of high-quality speech plus their corresponding transcriptions are available. However, laborious paired data collection processes prevent many institutes from building multi-speaker TTS systems of great performance. In this work, we propose a semi-supervised learning approach for multi-speaker TTS. A multi-speaker TTS model can learn from the untranscribed audio via the proposed encoder-decoder framework with discrete speech representation. The experiment results demonstrate that with only an hour of paired speech data, no matter the paired data is from multiple speakers or a single speaker, the proposed model can generate intelligible speech in different voices. We found the model can benefit from the proposed semi-supervised learning approach even when part of the unpaired speech data is noisy. In addition, our analysis reveals that different speaker characteristics of the paired

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keytu2020semi

Related papers

• Semi-supervised Training For Improving Data Efficiency In End-to-end Speech Synthesis (2018)13.28
• DQR-TTS: Semi-supervised Text-to-speech Synthesis With Dynamic Quantized Representation (2023)2.26
• Training Multi-speaker Neural Text-to-speech Systems Using Speaker-imbalanced Speech Corpora (2019)8.09
• Boosting Multi-speaker Expressive Speech Synthesis With Semi-supervised Contrastive Learning (2023)5.24
• Parrottts: Text-to-speech Synthesis By Exploiting Self-supervised Representations (2023)0.00
• Transfer Learning From Speaker Verification To Multispeaker Text-to-speech Synthesis (2018)0.00
• Extending Multilingual Speech Synthesis To 100+ Languages Without Transcribed Data (2024)7.16
• Learning Speaker Embedding From Text-to-speech (2020)5.84