MAST: Multiscale Audio Spectrogram Transformers

Abstract

We present Multiscale Audio Spectrogram Transformer (MAST) for audio classification, which brings the concept of multiscale feature hierarchies to the Audio Spectrogram Transformer (AST). Given an input audio spectrogram, we first patchify and project it into an initial temporal resolution and embedding dimension, post which the multiple stages in MAST progressively expand the embedding dimension while reducing the temporal resolution of the input. We use a pyramid structure that allows early layers of MAST operating at a high temporal resolution but low embedding space to model simple low-level acoustic information and deeper temporally coarse layers to model high-level acoustic information with high-dimensional embeddings. We also extend our approach to present a new Self-Supervised Learning (SSL) method called SS-MAST, which calculates a symmetric contrastive loss between latent representations from a student and a teacher encoder, leveraging patch-drop, a novel audio augmentation a

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyghosh2022mast

Related papers

• SSAST: Self-supervised Audio Spectrogram Transformer (2021)17.61
• Audio Mamba: Bidirectional State Space Model For Audio Representation Learning (2024)11.58
• Asit: Local-global Audio Spectrogram Vision Transformer For Event Classification (2022)8.35
• Elasticast: An Audio Spectrogram Transformer For All Length And Resolutions (2024)3.58
• Audio Mamba: Selective State Spaces For Self-supervised Audio Representations (2024)9.23
• Multi-class-token Transformer For Multitask Self-supervised Music Information Retrieval (2025)0.00
• Axlstms: Learning Self-supervised Audio Representations With Xlstms (2024)2.26
• SSAMBA: Self-supervised Audio Representation Learning With Mamba State Space Model (2024)0.00