SALTTS: Leveraging Self-supervised Speech Representations For Improved Text-to-speech Synthesis

Abstract

While FastSpeech2 aims to integrate aspects of speech such as pitch, energy, and duration as conditional inputs, it still leaves scope for richer representations. As a part of this work, we leverage representations from various Self-Supervised Learning (SSL) models to enhance the quality of the synthesized speech. In particular, we pass the FastSpeech2 encoder's length-regulated outputs through a series of encoder layers with the objective of reconstructing the SSL representations. In the SALTTS-parallel implementation, the representations from this second encoder are used for an auxiliary reconstruction loss with the SSL features. The SALTTS-cascade implementation, however, passes these representations through the decoder in addition to having the reconstruction loss. The richness of speech characteristics from the SSL features reflects in the output speech quality, with the objective and subjective evaluation measures of the proposed approach outperforming the baseline FastSpeech2.

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keysivaguru2023saltts

Related papers

• On The Use Of Self-supervised Speech Representations In Spontaneous Speech Synthesis (2023)0.00
• Analytic Study Of Text-free Speech Synthesis For Raw Audio Using A Self-supervised Learning Model (2024)0.00
• Low-resource Self-supervised Learning With Ssl-enhanced TTS (2023)0.00
• Efficient Infusion Of Self-supervised Representations In Automatic Speech Recognition (2024)0.00
• The Efficacy Of Self-supervised Speech Models For Audio Representations (2022)0.00
• Target Speech Extraction With Pre-trained Self-supervised Learning Models (2024)9.41
• Robustl2s: Speaker-specific Lip-to-speech Synthesis Exploiting Self-supervised Representations (2023)4.52
• Fusion Of Discrete Representations And Self-augmented Representations For Multilingual Automatic Speech Recognition (2024)2.26