Less Forgetting For Better Generalization: Exploring Continual-learning Fine-tuning Methods For Speech Self-supervised Representations

Abstract

Despite being trained on massive and diverse datasets, speech self-supervised encoders are generally used for downstream purposes as mere frozen feature extractors or model initializers before fine-tuning. The former severely limits the exploitation of large encoders, while the latter hurts the robustness acquired during pretraining, especially in low-resource scenarios. This work explores middle-ground solutions, conjecturing that reducing the forgetting of the self-supervised task during the downstream fine-tuning leads to better generalization. To prove this, focusing on speech recognition, we benchmark different continual-learning approaches during fine-tuning and show that they improve both in-domain and out-of-domain generalization abilities. Relative performance gains reach 15.7% and 22.5% with XLSR used as the encoder on two English and Danish speech recognition tasks. Further probing experiments show that these gains are indeed linked to less forgetting.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzaiem2024less

Related papers

• Fine-tuning Strategies For Faster Inference Using Speech Self-supervised Models: A Comparative Study (2023)8.35
• Self-supervised Rewiring Of Pre-trained Speech Encoders: Towards Faster Fine-tuning With Less Labels In Speech Processing (2022)3.58
• Continual Learning For Monolingual End-to-end Automatic Speech Recognition (2021)7.16
• Updating Only Encoders Prevents Catastrophic Forgetting Of End-to-end ASR Models (2022)5.24
• Learning Problem-agnostic Speech Representations From Multiple Self-supervised Tasks (2019)15.54
• Feature Normalization For Fine-tuning Self-supervised Models In Speech Enhancement (2023)2.26
• Efficient Personalized Speech Enhancement Through Self-supervised Learning (2021)10.21
• A Dual Task Learning Approach To Fine-tune A Multilingual Semantic Speech Encoder For Spoken Language Understanding (2024)0.00