Lightweight And High-fidelity End-to-end Text-to-speech With Multi-band Generation And Inverse Short-time Fourier Transform

Abstract

We propose a lightweight end-to-end text-to-speech model using multi-band generation and inverse short-time Fourier transform. Our model is based on VITS, a high-quality end-to-end text-to-speech model, but adopts two changes for more efficient inference: 1) the most computationally expensive component is partially replaced with a simple inverse short-time Fourier transform, and 2) multi-band generation, with fixed or trainable synthesis filters, is used to generate waveforms. Unlike conventional lightweight models, which employ optimization or knowledge distillation separately to train two cascaded components, our method enjoys the full benefits of end-to-end optimization. Experimental results show that our model synthesized speech as natural as that synthesized by VITS, while achieving a real-time factor of 0.066 on an Intel Core i7 CPU, 4.1 times faster than VITS. Moreover, a smaller version of the model significantly outperformed a lightweight baseline model with respect to both na

Stats

• citations16
• S2 citations—
• github stars468
• HF likes0
• heat score14.57
• arxiv keykawamura2022lightweight

Related papers

• FLY-TTS: Fast, Lightweight And High-quality End-to-end Text-to-speech Synthesis (2024)0.00
• Quickvc: Any-to-many Voice Conversion Using Inverse Short-time Fourier Transform For Faster Conversion (2023)0.00
• VITS2: Improving Quality And Efficiency Of Single-stage Text-to-speech With Adversarial Learning And Architecture Design (2023)12.40
• Period VITS: Variational Inference With Explicit Pitch Modeling For End-to-end Emotional Speech Synthesis (2022)8.60
• PITS: Variational Pitch Inference Without Fundamental Frequency For End-to-end Pitch-controllable TTS (2023)4.90
• Conditional Variational Autoencoder With Adversarial Learning For End-to-end Text-to-speech (2021)0.00
• JETS: Jointly Training Fastspeech2 And Hifi-gan For End To End Text To Speech (2022)12.10
• Fast And Small Footprint Hybrid Hmm-hifigan Based System For Speech Synthesis In Indian Languages (2023)0.00