Seeing Through The Conversation: Audio-visual Speech Separation Based On Diffusion Model

Abstract

The objective of this work is to extract target speaker's voice from a mixture of voices using visual cues. Existing works on audio-visual speech separation have demonstrated their performance with promising intelligibility, but maintaining naturalness remains a challenge. To address this issue, we propose AVDiffuSS, an audio-visual speech separation model based on a diffusion mechanism known for its capability in generating natural samples. For an effective fusion of the two modalities for diffusion, we also propose a cross-attention-based feature fusion mechanism. This mechanism is specifically tailored for the speech domain to integrate the phonetic information from audio-visual correspondence in speech generation. In this way, the fusion process maintains the high temporal resolution of the features, without excessive computational requirements. We demonstrate that the proposed framework achieves state-of-the-art results on two benchmarks, including VoxCeleb2 and LRS3, producing sp

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keylee2023seeing

Related papers

• Audio-visual Speech Separation Based On Joint Feature Representation With Cross-modal Attention (2022)0.00
• Separate And Diffuse: Using A Pretrained Diffusion Model For Improving Source Separation (2023)0.00
• Diff-foley: Synchronized Video-to-audio Synthesis With Latent Diffusion Models (2023)0.00
• An Overview Of Deep-learning-based Audio-visual Speech Enhancement And Separation (2020)18.31
• Contrastive Conditional Latent Diffusion For Audio-visual Segmentation (2023)10.29
• Diffusion-based Unsupervised Audio-visual Speech Enhancement (2024)4.52
• Av-link: Temporally-aligned Diffusion Features For Cross-modal Audio-video Generation (2024)0.00
• Looking Into Your Speech: Learning Cross-modal Affinity For Audio-visual Speech Separation (2021)11.67