SSAST: Self-supervised Audio Spectrogram Transformer

Abstract

Recently, neural networks based purely on self-attention, such as the Vision Transformer (ViT), have been shown to outperform deep learning models constructed with convolutional neural networks (CNNs) on various vision tasks, thus extending the success of Transformers, which were originally developed for language processing, to the vision domain. A recent study showed that a similar methodology can also be applied to the audio domain. Specifically, the Audio Spectrogram Transformer (AST) achieves state-of-the-art results on various audio classification benchmarks. However, pure Transformer models tend to require more training data compared to CNNs, and the success of the AST relies on supervised pretraining that requires a large amount of labeled data and a complex training pipeline, thus limiting the practical usage of AST. This paper focuses on audio and speech classification, and aims to reduce the need for large amounts of labeled data for AST by leveraging self-supervised learni

Stats

• citations222
• S2 citations—
• github stars0
• HF likes0
• heat score17.61
• arxiv keygong2021ssast

Related papers

• Asit: Local-global Audio Spectrogram Vision Transformer For Event Classification (2022)8.35
• Elasticast: An Audio Spectrogram Transformer For All Length And Resolutions (2024)3.58
• Axlstms: Learning Self-supervised Audio Representations With Xlstms (2024)2.26
• Audio Mamba: Bidirectional State Space Model For Audio Representation Learning (2024)11.58
• MAST: Multiscale Audio Spectrogram Transformers (2022)4.52
• Adapter Incremental Continual Learning Of Efficient Audio Spectrogram Transformers (2023)6.34
• Audio Mamba: Selective State Spaces For Self-supervised Audio Representations (2024)9.23
• Multi-class-token Transformer For Multitask Self-supervised Music Information Retrieval (2025)0.00