Adversarial Speaker-consistency Learning Using Untranscribed Speech Data For Zero-shot Multi-speaker Text-to-speech

Abstract

Several recently proposed text-to-speech (TTS) models achieved to generate the speech samples with the human-level quality in the single-speaker and multi-speaker TTS scenarios with a set of pre-defined speakers. However, synthesizing a new speaker's voice with a single reference audio, commonly known as zero-shot multi-speaker text-to-speech (ZSM-TTS), is still a very challenging task. The main challenge of ZSM-TTS is the speaker domain shift problem upon the speech generation of a new speaker. To mitigate this problem, we propose adversarial speaker-consistency learning (ASCL). The proposed method first generates an additional speech of a query speaker using the external untranscribed datasets at each training iteration. Then, the model learns to consistently generate the speech sample of the same speaker as the corresponding speaker embedding vector by employing an adversarial learning scheme. The experimental results show that the proposed method is effective compared to the baseli

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keychoi2022adversarial

Related papers

• ZMM-TTS: Zero-shot Multilingual And Multispeaker Speech Synthesis Conditioned On Self-supervised Discrete Speech Representations (2023)10.35
• Nnspeech: Speaker-guided Conditional Variational Autoencoder For Zero-shot Multi-speaker Text-to-speech (2022)9.59
• Semi-supervised Learning For Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation (2020)5.24
• Generalizable Zero-shot Speaker Adaptive Speech Synthesis With Disentangled Representations (2023)6.34
• Speaker Consistency Loss And Step-wise Optimization For Semi-supervised Joint Training Of TTS And ASR Using Unpaired Text Data (2022)0.00
• ASRRL-TTS: Agile Speaker Representation Reinforcement Learning For Text-to-speech Speaker Adaptation (2024)0.00
• Enhancing Zero-shot Multi-speaker TTS With Negated Speaker Representations (2024)3.58
• Data Efficient Voice Cloning From Noisy Samples With Domain Adversarial Training (2020)9.92