Adapter Incremental Continual Learning Of Efficient Audio Spectrogram Transformers

Abstract

Continual learning involves training neural networks incrementally for new tasks while retaining the knowledge of previous tasks. However, efficiently fine-tuning the model for sequential tasks with minimal computational resources remains a challenge. In this paper, we propose Task Incremental Continual Learning (TI-CL) of audio classifiers with both parameter-efficient and compute-efficient Audio Spectrogram Transformers (AST). To reduce the trainable parameters without performance degradation for TI-CL, we compare several Parameter Efficient Transfer (PET) methods and propose AST with Convolutional Adapters for TI-CL, which has less than 5% of trainable parameters of the fully fine-tuned counterparts. To reduce the computational complexity, we introduce a novel Frequency-Time factorized Attention (FTA) method that replaces the traditional self-attention in transformers for audio spectrograms. FTA achieves competitive performance with only a factor of the computations required by Glob

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keyselvaraj2023adapter

Related papers

• Integrated Parameter-efficient Tuning For General-purpose Audio Models (2022)0.00
• SSAST: Self-supervised Audio Spectrogram Transformer (2021)17.61
• Asit: Local-global Audio Spectrogram Vision Transformer For Event Classification (2022)8.35
• Elasticast: An Audio Spectrogram Transformer For All Length And Resolutions (2024)3.58
• Leveraging Parameter-efficient Transfer Learning For Multi-lingual Text-to-speech Adaptation (2024)0.00
• Efficient Large-scale Audio Tagging Via Transformer-to-cnn Knowledge Distillation (2022)17.68
• Parameter-efficient Transfer Learning Of Pre-trained Transformer Models For Speaker Verification Using Adapters (2022)0.00
• Input-independent Attention Weights Are Expressive Enough: A Study Of Attention In Self-supervised Audio Transformers (2020)0.00