Audio-visual Speech Separation And Dereverberation With A Two-stage Multimodal Network

Abstract

Background noise, interfering speech and room reverberation frequently distort target speech in real listening environments. In this study, we address joint speech separation and dereverberation, which aims to separate target speech from background noise, interfering speech and room reverberation. In order to tackle this fundamentally difficult problem, we propose a novel multimodal network that exploits both audio and visual signals. The proposed network architecture adopts a two-stage strategy, where a separation module is employed to attenuate background noise and interfering speech in the first stage and a dereverberation module to suppress room reverberation in the second stage. The two modules are first trained separately, and then integrated for joint training, which is based on a new multi-objective loss function. Our experimental results show that the proposed multimodal network yields consistently better objective intelligibility and perceptual quality than several one-stage

Stats

• citations45
• S2 citations—
• github stars0
• HF likes0
• heat score12.47
• arxiv keytan2019audio

Related papers

• Audio-visual Multi-channel Speech Separation, Dereverberation And Recognition (2022)6.77
• A Multi-stage Triple-path Method For Speech Separation In Noisy And Reverberant Environments (2023)2.26
• Audio-visual Speech Separation Based On Joint Feature Representation With Cross-modal Attention (2022)0.00
• SADDEL: Joint Speech Separation And Denoising Model Based On Multitask Learning (2020)0.00
• Two-stage Model And Optimal SI-SNR For Monaural Multi-speaker Speech Separation In Noisy Environment (2020)0.00
• Time Domain Audio Visual Speech Separation (2019)14.62
• Convolutive Prediction For Monaural Speech Dereverberation And Noisy-reverberant Speaker Separation (2021)11.39
• Multi-channel Multi-frame ADL-MVDR For Target Speech Separation (2020)0.00