Weight Averaging: A Simple Yet Effective Method To Overcome Catastrophic Forgetting In Automatic Speech Recognition

Abstract

Adapting a trained Automatic Speech Recognition (ASR) model to new tasks results in catastrophic forgetting of old tasks, limiting the model's ability to learn continually and to be extended to new speakers, dialects, languages, etc. Focusing on End-to-End ASR, in this paper, we propose a simple yet effective method to overcome catastrophic forgetting: weight averaging. By simply taking the average of the previous and the adapted model, our method achieves high performance on both the old and new tasks. It can be further improved by introducing a knowledge distillation loss during the adaptation. We illustrate the effectiveness of our method on both monolingual and multilingual ASR. In both cases, our method strongly outperforms all baselines, even in its simplest form.

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyeeckt2022weight

Related papers

• Continual Learning For Monolingual End-to-end Automatic Speech Recognition (2021)7.16
• Rehearsal-free Online Continual Learning For Automatic Speech Recognition (2023)5.24
• Updating Only Encoders Prevents Catastrophic Forgetting Of End-to-end ASR Models (2022)5.24
• Residual Adapters For Parameter-efficient ASR Adaptation To Atypical And Accented Speech (2021)10.74
• Reducing Geographic Disparities In Automatic Speech Recognition Via Elastic Weight Consolidation (2022)2.26
• Unsupervised Online Continual Learning For Automatic Speech Recognition (2024)4.52
• Bayesian Parameter-efficient Fine-tuning For Overcoming Catastrophic Forgetting (2024)0.00
• Continual Speaker Adaptation For Text-to-speech Synthesis (2021)0.00