Siamese Vision Transformers Are Scalable Audio-visual Learners

Abstract

Traditional audio-visual methods rely on independent audio and visual backbones, which is costly and not scalable. In this work, we investigate using an audio-visual siamese network (AVSiam) for efficient and scalable audio-visual pretraining. Our framework uses a single shared vision transformer backbone to process audio and visual inputs, improving its parameter efficiency, reducing the GPU memory footprint, and allowing us to scale our method to larger datasets and model sizes. We pretrain our model using a contrastive audio-visual matching objective with a multi-ratio random masking scheme, which enables our model to process larger audio-visual instance batches, helpful for contrastive learning. Unlike prior audio-visual methods, our method can robustly handle audio, visual, and audio-visual inputs with a single shared ViT backbone. Furthermore, despite using the shared backbone for both modalities, AVSiam achieves competitive or even better results than prior methods on AudioSet a

Stats

• citations4
• S2 citations—
• github stars12
• HF likes0
• heat score7.47
• arxiv keylin2024siamese

Related papers

• Sequential Contrastive Audio-visual Learning (2024)5.84
• GMS-CAVP: Improving Audio-video Correspondence With Multi-scale Contrastive And Generative Pretraining (2026)0.00
• CAVL: Learning Contrastive And Adaptive Representations Of Vision And Language (2023)0.00
• Billion-scale Pretraining With Vision Transformers For Multi-task Visual Representations (2021)9.23
• Contrastive Audio-visual Masked Autoencoder (2022)4.93
• Objects That Sound (2017)0.00
• Thinking Fast And Slow: Efficient Text-to-visual Retrieval With Transformers (2021)15.16
• See, Hear, And Read: Deep Aligned Representations (2017)0.00