General-purpose Speech Representation Learning Through A Self-supervised Multi-granularity Framework

Abstract

This paper presents a self-supervised learning framework, named MGF, for general-purpose speech representation learning. In the design of MGF, speech hierarchy is taken into consideration. Specifically, we propose to use generative learning approaches to capture fine-grained information at small time scales and use discriminative learning approaches to distill coarse-grained or semantic information at large time scales. For phoneme-scale learning, we borrow idea from the masked language model but tailor it for the continuous speech signal by replacing classification loss with a contrastive loss. We corroborate our design by evaluating MGF representation on various downstream tasks, including phoneme classification, speaker classification, speech recognition, and emotion classification. Experiments verify that training at different time scales needs different training targets and loss functions, which in general complement each other and lead to a better performance.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhao2021general

Related papers

• Learning Problem-agnostic Speech Representations From Multiple Self-supervised Tasks (2019)15.54
• An Unsupervised Autoregressive Model For Speech Representation Learning (2019)17.26
• Hierarchical Multi-grained Generative Model For Expressive Speech Synthesis (2020)8.60
• Universal Paralinguistic Speech Representations Using Self-supervised Conformers (2021)10.48
• Efficient Personalized Speech Enhancement Through Self-supervised Learning (2021)10.21
• Less Forgetting For Better Generalization: Exploring Continual-learning Fine-tuning Methods For Speech Self-supervised Representations (2024)0.00
• Multi-resolution Hubert: Multi-resolution Speech Self-supervised Learning With Masked Unit Prediction (2023)0.00
• Analysing Discrete Self Supervised Speech Representation For Spoken Language Modeling (2023)12.86