Self-supervised Learning For Speech Recognition With Intermediate Layer Supervision

Abstract

Recently, pioneer work finds that speech pre-trained models can solve full-stack speech processing tasks, because the model utilizes bottom layers to learn speaker-related information and top layers to encode content-related information. Since the network capacity is limited, we believe the speech recognition performance could be further improved if the model is dedicated to audio content information learning. To this end, we propose Intermediate Layer Supervision for Self-Supervised Learning (ILS-SSL), which forces the model to concentrate on content information as much as possible by adding an additional SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set show that our method outperforms HuBERT significantly, which achieves a 23.5%/11.6% relative word error rate reduction in the w/o language model setting for base/large models. Detailed analysis shows the bottom layers of our model have a better correlation with phonetic units, which is consistent with our

Stats

• citations17
• S2 citations—
• github stars0
• HF likes0
• heat score9.41
• arxiv keywang2021self

Related papers

• Unispeech-sat: Universal Speech Representation Learning With Speaker Aware Pre-training (2021)0.00
• Pushing The Limits Of Unsupervised Unit Discovery For SSL Speech Representation (2023)6.34
• Efficient Infusion Of Self-supervised Representations In Automatic Speech Recognition (2024)0.00
• Automatic Pronunciation Assessment Using Self-supervised Speech Representation Learning (2022)0.00
• Progressive Residual Extraction Based Pre-training For Speech Representation Learning (2024)0.00
• Weakly-supervised Speech Pre-training: A Case Study On Target Speech Recognition (2023)8.09
• Multi-resolution Hubert: Multi-resolution Speech Self-supervised Learning With Masked Unit Prediction (2023)0.00
• Investigating Self-supervised Learning For Speech Enhancement And Separation (2022)13.44