Cobert: Self-supervised Speech Representation Learning Through Code Representation Learning

Abstract

Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes. We propose the Code BERT (CoBERT) approach for self-supervised speech representation learning. The idea is to convert an utterance to a sequence of discrete codes, and perform code representation learning, where we predict the code representations based on a masked view of the original speech input. Unlike the prior self-distillation approaches of which the teacher and the student are of the same modality, our target model predicts representations from a different modality. CoBERT outperforms the most recent state-of-the-art performance on the ASR task and brings significant improvements on the SUPERB speech translation (ST) task. Our code and models are released at https://github.com/mct10/CoBERT.

Stats

• citations0
• S2 citations—
• github stars48
• HF likes0
• heat score3.38
• arxiv keymeng2022cobert

Code

• mct10/CoBERT—

Related papers

• Hubert: Self-supervised Speech Representation Learning By Masked Prediction Of Hidden Units (2021)25.30
• Spatial Hubert: Self-supervised Spatial Speech Representation Learning For A Single Talker From Multi-channel Audio (2023)0.00
• Supervision-guided Codebooks For Masked Prediction In Speech Pre-training (2022)7.81
• Audio ALBERT: A Lite BERT For Self-supervised Learning Of Audio Representation (2020)15.54
• Unispeech-sat: Universal Speech Representation Learning With Speaker Aware Pre-training (2021)0.00
• Contrastive Separative Coding For Self-supervised Representation Learning (2021)0.00
• Contentvec: An Improved Self-supervised Speech Representation By Disentangling Speakers (2022)0.00
• Ms-hubert: Mitigating Pre-training And Inference Mismatch In Masked Language Modelling Methods For Learning Speech Representations (2024)4.52