Efficient Large-scale Audio Tagging Via Transformer-to-cnn Knowledge Distillation

Abstract

Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of .483 mAP on AudioSet. Source Code available at: https://github.com/fschmid56/EfficientAT

Stats

• citations47
• S2 citations—
• github stars341
• HF likes0
• heat score17.68
• arxiv keyschmid2022efficient

Code

• fschmid56/EfficientAT—

Related papers

• Dynamic Convolutional Neural Networks As Efficient Pre-trained Audio Models (2023)0.00
• Efficient Training Of Audio Transformers With Patchout (2021)22.11
• Streaming Audio Transformers For Online Audio Tagging (2023)3.48
• SSAST: Self-supervised Audio Spectrogram Transformer (2021)17.61
• Asit: Local-global Audio Spectrogram Vision Transformer For Event Classification (2022)8.35
• Audiomer: A Convolutional Transformer For Keyword Spotting (2021)0.00
• Taming Data And Transformers For Audio Generation (2024)0.00
• Adapter Incremental Continual Learning Of Efficient Audio Spectrogram Transformers (2023)6.34