Investigating The 'autoencoder Behavior' In Speech Self-supervised Models: A Focus On Hubert's Pretraining

Abstract

Self-supervised learning has shown great success in Speech Recognition. However, it has been observed that finetuning all layers of the learned model leads to lower performance compared to resetting top layers. This phenomenon is attributed to the ''autoencoder'' behavior: top layers contain information closer to the input and are less suitable for tasks that require linguistic information, such as Speech Recognition.To better our understanding of this behavior, we propose to study the evolution of high-level information within the model during pretraining. We focus on the HuBERT model, which exhibits a less pronounced ''autoencoder'' behavior. By experimentally exploring various factors that may have an impact, we aim to improve the training procedure and enhance the top layers of HuBERT for high-level tasks.Furthermore, our experiments demonstrate that these improvements in the training procedure result in faster convergence and competitive performance on downstream tasks.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvielzeuf2024investigating

Related papers

• Fast-hubert: An Efficient Training Framework For Self-supervised Speech Representation Learning (2023)0.00
• Hubert-ee: Early Exiting Hubert For Efficient Speech Recognition (2022)6.77
• Selective Hubert: Self-supervised Pre-training For Target Speaker In Clean And Mixture Speech (2023)7.81
• Ms-hubert: Mitigating Pre-training And Inference Mismatch In Masked Language Modelling Methods For Learning Speech Representations (2024)4.52
• Multi-resolution Hubert: Multi-resolution Speech Self-supervised Learning With Masked Unit Prediction (2023)0.00
• Self-supervised Learning For Speech Recognition With Intermediate Layer Supervision (2021)9.41
• Melhubert: A Simplified Hubert On Mel Spectrograms (2022)3.70
• Progressive Residual Extraction Based Pre-training For Speech Representation Learning (2024)0.00