Bayesian Parameter-efficient Fine-tuning For Overcoming Catastrophic Forgetting
2024 · Haolin Chen, Philip N. Garner
Abstract
We are motivated primarily by the adaptation of text-to-speech synthesis models; however we argue that more generic parameter-efficient fine-tuning (PEFT) is an appropriate framework to do such adaptation. Nevertheless, catastrophic forgetting remains an issue with PEFT, damaging the pre-trained model's inherent capabilities. We demonstrate that existing Bayesian learning techniques can be applied to PEFT to prevent catastrophic forgetting as long as the parameter shift of the fine-tuned layers can be calculated differentiably. In a principled series of experiments on language modeling and speech synthesis tasks, we utilize established Laplace approximations, including diagonal and Kronecker-factored approaches, to regularize PEFT with the low-rank adaptation (LoRA) and compare their performance in pre-training knowledge preservation. Our results demonstrate that catastrophic forgetting can be overcome by our methods without degrading the fine-tuning performance, and using the Kronecke
Authors
(none)
Tags
Stats
Related papers
- Updating Only Encoders Prevents Catastrophic Forgetting Of End-to-end ASR Models (2022)5.24
- Efficient Emotion And Speaker Adaptation In Llm-based TTS Via Characteristic-specific Partial Fine-tuning (2025)0.00
- Leveraging Parameter-efficient Transfer Learning For Multi-lingual Text-to-speech Adaptation (2024)0.00
- Continual Speaker Adaptation For Text-to-speech Synthesis (2021)0.00
- Understanding Textual Capability Degradation In Speech Llms Via Parameter Importance Analysis (2025)0.00
- Residual Adapters For Parameter-efficient ASR Adaptation To Atypical And Accented Speech (2021)10.74
- Weight Averaging: A Simple Yet Effective Method To Overcome Catastrophic Forgetting In Automatic Speech Recognition (2022)6.34
- Less Forgetting For Better Generalization: Exploring Continual-learning Fine-tuning Methods For Speech Self-supervised Representations (2024)0.00