Evaluating Parameter-efficient Transfer Learning Approaches On SURE Benchmark For Speech Understanding
2023 · Yingting Li, Ambuj Mehrish, Shuai Zhao, et al.
Abstract
Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, and HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech-processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outpe
Authors
(none)
Tags
Stats
Related papers
- Parameter-efficient Transfer Learning Of Pre-trained Transformer Models For Speaker Verification Using Adapters (2022)0.00
- Elp-adapters: Parameter Efficient Adapter Tuning For Various Speech Processing Tasks (2024)7.81
- Exploring Efficient-tuning Methods In Self-supervised Speech Models (2022)10.07
- Efficient Adapter Tuning Of Pre-trained Speech Models For Automatic Speaker Verification (2024)0.00
- Adapter-based Extension Of Multi-speaker Text-to-speech Model For New Speakers (2022)6.77
- An Adapter-based Unified Model For Multiple Spoken Language Processing Tasks (2024)0.00
- Unipet-spk: A Unified Framework For Parameter-efficient Tuning Of Pre-trained Speech Models For Robust Speaker Verification (2025)4.52
- Efficient Adapter Transfer Of Self-supervised Speech Models For Automatic Speech Recognition (2022)12.68