Sequential Contrastive Audio-visual Learning

Abstract

Contrastive learning has emerged as a powerful technique in audio-visual representation learning, leveraging the natural co-occurrence of audio and visual modalities in webscale video datasets. However, conventional contrastive audio-visual learning (CAV) methodologies often rely on aggregated representations derived through temporal aggregation, neglecting the intrinsic sequential nature of the data. This oversight raises concerns regarding the ability of standard approaches to capture and utilize fine-grained information within sequences. In response to this limitation, we propose sequential contrastive audiovisual learning (SCAV), which contrasts examples based on their non-aggregated representation space using multidimensional sequential distances. Audio-visual retrieval experiments with the VGGSound and Music datasets demonstrate the effectiveness of SCAV, with up to 3.5x relative improvements in recall against traditional aggregation-based contrastive learning and other previousl

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keytsiamas2024sequential

Related papers

• Contrastive Audio-visual Masked Autoencoder (2022)4.93
• CAVL: Learning Contrastive And Adaptive Representations Of Vision And Language (2023)0.00
• GMS-CAVP: Improving Audio-video Correspondence With Multi-scale Contrastive And Generative Pretraining (2026)0.00
• Contrastive Audio-language Learning For Music (2022)0.00
• Siamese Vision Transformers Are Scalable Audio-visual Learners (2024)7.47
• Temporal Context Aggregation For Video Retrieval With Contrastive Learning (2020)13.23
• Normalized Contrastive Learning For Text-video Retrieval (2022)6.77
• Quality Over Quantity? Llm-based Curation For A Data-efficient Audio-video Foundation Model (2025)0.00