Speech Resynthesis From Discrete Disentangled Self-supervised Representations

Abstract

We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (for both resynthesis and voice conversion), recordings' intelligibility, and overall quality using subjective human evaluation. Lastly, we demonstrate how these representations can be used for an ultra-lightweight speech codec. Using the obtained representations, we can get to a rate of 365 bits per second while providing better speech quality than the baseline methods. Audio samples can be found u

Stats

• citations146
• S2 citations—
• github stars0
• HF likes0
• heat score16.25
• arxiv keypolyak2021speech

Related papers

• ACE-VC: Adaptive And Controllable Voice Conversion Using Explicitly Disentangled Self-supervised Speech Representations (2023)0.00
• Disentangling Prosody Representations With Unsupervised Speech Reconstruction (2022)0.00
• Learning Disentangled Speech Representations (2023)0.00
• Perception Of Prosodic Variation For Speech Synthesis Using An Unsupervised Discrete Representation Of F0 (2020)7.81
• Controllable Speech Synthesis By Learning Discrete Phoneme-level Prosodic Representations (2022)6.34
• Robust Disentangled Variational Speech Representation Learning For Zero-shot Voice Conversion (2022)10.97
• EXPRESSO: A Benchmark And Analysis Of Discrete Expressive Speech Resynthesis (2023)0.00
• Codec-asr: Training Performant Automatic Speech Recognition Systems With Discrete Speech Representations (2024)6.77