Audio-visual Multi-channel Speech Separation, Dereverberation And Recognition

Abstract

Despite the rapid advance of automatic speech recognition (ASR) technologies, accurate recognition of cocktail party speech characterised by the interference from overlapping speakers, background noise and room reverberation remains a highly challenging task to date. Motivated by the invariance of visual modality to acoustic signal corruption, audio-visual speech enhancement techniques have been developed, although predominantly targeting overlapping speech separation and recognition tasks. In this paper, an audio-visual multi-channel speech separation, dereverberation and recognition approach featuring a full incorporation of visual information into all three stages of the system is proposed. The advantage of the additional visual modality over using audio only is demonstrated on two neural dereverberation approaches based on DNN-WPE and spectral mapping respectively. The learning cost function mismatch between the separation and dereverberation models and their integration with the b

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keyli2022audio

Related papers

• Audio-visual Speech Separation And Dereverberation With A Two-stage Multimodal Network (2019)12.47
• An Overview Of Deep-learning-based Audio-visual Speech Enhancement And Separation (2020)18.31
• Audio-visual Speech Separation Based On Joint Feature Representation With Cross-modal Attention (2022)0.00
• Time Domain Audio Visual Speech Separation (2019)14.62
• Speaker-targeted Audio-visual Models For Speech Recognition In Cocktail-party Environments (2019)0.00
• Robust End-to-end Deep Audiovisual Speech Recognition (2016)0.00
• Audio-visual Target Speaker Enhancement On Multi-talker Environment Using Event-driven Cameras (2019)8.09
• Joint Speaker Features Learning For Audio-visual Multichannel Speech Separation And Recognition (2024)0.00