Contrastive Audio-visual Masked Autoencoder

Abstract

In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task. Code and pretrained models are at https://github.com/yuangongnd/cav-mae.

Stats

• citations0
• S2 citations—
• github stars290
• HF likes0
• heat score4.93
• arxiv keygong2022contrastive

Code

• yuangongnd/cav-mae—

Related papers

• Sequential Contrastive Audio-visual Learning (2024)5.84
• GMS-CAVP: Improving Audio-video Correspondence With Multi-scale Contrastive And Generative Pretraining (2026)0.00
• Variational Autoencoder With CCA For Audio-visual Cross-modal Retrieval (2021)9.92
• VLMAE: Vision-language Masked Autoencoder (2022)0.00
• Cot-mae V2: Contextual Masked Auto-encoder With Multi-view Modeling For Passage Retrieval (2023)0.00
• Masked Contrastive Pre-training For Efficient Video-text Retrieval (2022)5.84
• Exploring Masked Autoencoders For Sensor-agnostic Image Retrieval In Remote Sensing (2024)10.74
• Video And Audio Are Images: A Cross-modal Mixer For Original Data On Video-audio Retrieval (2023)7.16