Toneunit: A Speech Discretization Approach For Tonal Language Speech Synthesis

Abstract

Representing speech as discretized units has numerous benefits in supporting downstream spoken language processing tasks. However, the approach has been less explored in speech synthesis of tonal languages like Mandarin Chinese. Our preliminary experiments on Chinese speech synthesis reveal the issue of "tone shift", where a synthesized speech utterance contains correct base syllables but incorrect tones. To address the issue, we propose the ToneUnit framework, which leverages annotated data with tone labels as CTC supervision to learn tone-aware discrete speech units for Mandarin Chinese speech. Our findings indicate that the discrete units acquired through the TonUnit resolve the "tone shift" issue in synthesized Chinese speech and yield favorable results in English synthesis. Moreover, the experimental results suggest that finite scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit can work effectively even with minimal annotated data.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keytao2024toneunit

Related papers

• The Realization Of Tones In Spontaneous Spoken Taiwan Mandarin: A Corpus-based Survey And Theory-driven Computational Modeling (2025)0.00
• Do Discrete Self-supervised Representations Of Speech Capture Tone Distinctions? (2024)0.00
• A Comparison Of Modeling Units In Sequence-to-sequence Speech Recognition With The Transformer On Mandarin Chinese (2018)11.39
• End-to-end Mandarin Tone Classification With Short Term Context Information (2021)0.00
• Research On Modeling Units Of Transformer Transducer For Mandarin Speech Recognition (2020)0.00
• Mandarin Tone Modeling Using Recurrent Neural Networks (2017)0.00
• A Unified Sequence-to-sequence Front-end Model For Mandarin Text-to-speech Synthesis (2019)9.41
• Building A Mixed-lingual Neural TTS System With Only Monolingual Data (2019)0.00