Hubert: Self-supervised Speech Representation Learning By Masked Prediction Of Hidden Units

Abstract

Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBE

Stats

• citations2364
• S2 citations—
• github stars0
• HF likes0
• heat score25.30
• arxiv keyhsu2021hubert

Related papers

• Learning Audio-visual Speech Representation By Masked Multimodal Cluster Prediction (2022)5.99
• Spatial Hubert: Self-supervised Spatial Speech Representation Learning For A Single Talker From Multi-channel Audio (2023)0.00
• Multi-resolution Hubert: Multi-resolution Speech Self-supervised Learning With Masked Unit Prediction (2023)0.00
• Ms-hubert: Mitigating Pre-training And Inference Mismatch In Masked Language Modelling Methods For Learning Speech Representations (2024)4.52
• Pushing The Limits Of Unsupervised Unit Discovery For SSL Speech Representation (2023)6.34
• Unsupervised Accent Adaptation Through Masked Language Model Correction Of Discrete Self-supervised Speech Units (2023)4.52
• Selective Hubert: Self-supervised Pre-training For Target Speaker In Clean And Mixture Speech (2023)7.81
• Hubertopic: Enhancing Semantic Representation Of Hubert Through Self-supervision Utilizing Topic Model (2023)0.00