Unsupervised Learning Of Disentangled Speech Content And Style Representation

Abstract

We present an approach for unsupervised learning of speech representation disentangling contents and styles. Our model consists of: (1) a local encoder that captures per-frame information; (2) a global encoder that captures per-utterance information; and (3) a conditional decoder that reconstructs speech given local and global latent variables. Our experiments show that (1) the local latent variables encode speech contents, as reconstructed speech can be recognized by ASR with low word error rates (WER), even with a different global encoding; (2) the global latent variables encode speaker style, as reconstructed speech shares speaker identity with the source utterance of the global encoding. Additionally, we demonstrate an useful application from our pre-trained model, where we can train a speaker recognition model from the global latent variables and achieve high accuracy by fine-tuning with as few data as one label per speaker.

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keytjandra2020unsupervised

Related papers

• Contrastive Predictive Coding Supported Factorized Variational Autoencoder For Unsupervised Learning Of Disentangled Speech Representations (2020)8.09
• Speaker And Style Disentanglement Of Speech Based On Contrastive Predictive Coding Supported Factorized Variational Autoencoder (2024)2.26
• Fine-grained Style Modeling, Transfer And Prediction In Text-to-speech Synthesis Via Phone-level Content-style Disentanglement (2020)9.41
• Contentvec: An Improved Self-supervised Speech Representation By Disentangling Speakers (2022)0.00
• Self-supervised Context-aware Style Representation For Expressive Speech Synthesis (2022)6.34
• Unsupervised Style And Content Separation By Minimizing Mutual Information For Speech Synthesis (2020)9.59
• Disentangling Voice And Content With Self-supervision For Speaker Recognition (2023)2.26
• Learning Latent Representations For Style Control And Transfer In End-to-end Speech Synthesis (2018)0.00