Masked Autoencoders With Multi-window Local-global Attention Are Better Audio Learners

Abstract

In this work, we propose a Multi-Window Masked Autoencoder (MW-MAE) fitted with a novel Multi-Window Multi-Head Attention (MW-MHA) module that facilitates the modelling of local-global interactions in every decoder transformer block through attention heads of several distinct local and global windows. Empirical results on ten downstream audio tasks show that MW-MAEs consistently outperform standard MAEs in overall performance and learn better general-purpose audio representations, along with demonstrating considerably better scaling characteristics. Investigating attention distances and entropies reveals that MW-MAE encoders learn heads with broader local and global attention. Analyzing attention head feature representations through Projection Weighted Canonical Correlation Analysis (PWCCA) shows that attention heads with the same window sizes across the decoder layers of the MW-MAE learn correlated feature representations which enables each block to independently capture local and glo

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyadav2023masked

Related papers

• Contrastive Audio-visual Masked Autoencoder (2022)4.93
• A Multiscale Autoencoder (MSAE) Framework For End-to-end Neural Network Speech Enhancement (2023)6.34
• Self-supervised Learning Based Monaural Speech Enhancement With Multi-task Pre-training (2021)0.00
• A-JEPA: Joint-embedding Predictive Architecture Can Listen (2023)0.00
• Improving Dual-microphone Speech Enhancement By Learning Cross-channel Features With Multi-head Attention (2022)6.77
• Mowe-audio: Multitask Audiollms With Mixture Of Weak Encoders (2024)3.58
• Mfcca:multi-frame Cross-channel Attention For Multi-speaker ASR In Multi-party Meeting Scenario (2022)7.81
• Multi-window Data Augmentation Approach For Speech Emotion Recognition (2020)0.00