Contrastive Conditional Latent Diffusion For Audio-visual Segmentation

Abstract

We propose a contrastive conditional latent diffusion model for audio-visual segmentation (AVS) to thoroughly investigate the impact of audio, where the correlation between audio and the final segmentation map is modeled to guarantee the strong correlation between them. To achieve semantic-correlated representation learning, our framework incorporates a latent diffusion model. The diffusion model learns the conditional generation process of the ground-truth segmentation map, resulting in ground-truth aware inference during the denoising process at the test stage. As our model is conditional, it is vital to ensure that the conditional variable contributes to the model output. We thus extensively model the contribution of the audio signal by minimizing the density ratio between the conditional probability of the multimodal data, e.g. conditioned on the audio-visual data, and that of the unimodal data, e.g. conditioned on the audio data only. In this way, our latent diffusion model via de

Stats

• citations13
• S2 citations—
• github stars6
• HF likes0
• heat score10.29
• arxiv keymao2023contrastive

Related papers

• Seeing Through The Conversation: Audio-visual Speech Separation Based On Diffusion Model (2023)7.50
• Diff-foley: Synchronized Video-to-audio Synthesis With Latent Diffusion Models (2023)0.00
• Conditional Latent Diffusion-based Speech Enhancement Via Dual Context Learning (2025)10.81
• Diffusion Models As Masked Audio-video Learners (2023)0.00
• Diffusion-based Unsupervised Audio-visual Speech Enhancement (2024)4.52
• Av-link: Temporally-aligned Diffusion Features For Cross-modal Audio-video Generation (2024)0.00
• Audio Visual Segmentation Through Text Embeddings (2025)1.81
• Audiotoken: Adaptation Of Text-conditioned Diffusion Models For Audio-to-image Generation (2023)9.76