Supervision-guided Codebooks For Masked Prediction In Speech Pre-training

Abstract

Recently, masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. It usually requires a codebook obtained in an unsupervised way, making it less accurate and difficult to interpret. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance and also the pre-training efficiency, either through decoding with a hybrid ASR system to generate phoneme-level alignments (named PBERT), or performing clustering on the supervised speech features extracted from an end-to-end CTC model (named CTC clustering). Both the hybrid and CTC models are trained on the same small amount of labeled speech as used in fine-tuning. Experiments demonstrate significant superiority of our methods to various SSL and self-training baselines, with up to 17.0% relative WER reduction. Our pre-trained models also show good transferability in a non-ASR speech task.

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keywang2022supervision

Related papers

• Guided Contrastive Self-supervised Pre-training For Automatic Speech Recognition (2022)0.00
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019)0.00
• Pre-training Transformer Decoder For End-to-end ASR Model With Unpaired Speech Data (2022)13.47
• Analysing The Masked Predictive Coding Training Criterion For Pre-training A Speech Representation Model (2023)4.52
• Unsupervised Pre-training Of Bidirectional Speech Encoders Via Masked Reconstruction (2020)12.33
• Wav2code: Restore Clean Speech Representations Via Codebook Lookup For Noise-robust ASR (2023)8.35
• Joint Masked CPC And CTC Training For ASR (2020)8.60
• Weakly-supervised Speech Pre-training: A Case Study On Target Speech Recognition (2023)8.09