Audio-visual Target Speaker Enhancement On Multi-talker Environment Using Event-driven Cameras

Abstract

We propose a method to address audio-visual target speaker enhancement in multi-talker environments using event-driven cameras. State of the art audio-visual speech separation methods shows that crucial information is the movement of the facial landmarks related to speech production. However, all approaches proposed so far work offline, using frame-based video input, making it difficult to process an audio-visual signal with low latency, for online applications. In order to overcome this limitation, we propose the use of event-driven cameras and exploit compression, high temporal resolution and low latency, for low cost and low latency motion feature extraction, going towards online embedded audio-visual speech processing. We use the event-driven optical flow estimation of the facial landmarks as input to a stacked Bidirectional LSTM trained to predict an Ideal Amplitude Mask that is then used to filter the noisy audio, to obtain the audio signal of the target speaker. The presented ap

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keyarriandiaga2019audio

Related papers

• Face Landmark-based Speaker-independent Audio-visual Speech Enhancement In Multi-talker Environments (2018)12.40
• Multi-input Multi-output Target-speaker Voice Activity Detection For Unified, Flexible, And Robust Audio-visual Speaker Diarization (2024)0.00
• Speaker-targeted Audio-visual Models For Speech Recognition In Cocktail-party Environments (2019)0.00
• Audio-visual Speech Separation Based On Joint Feature Representation With Cross-modal Attention (2022)0.00
• Audio-visual Active Speaker Extraction For Sparsely Overlapped Multi-talker Speech (2023)7.50
• Audio-visual Multi-channel Speech Separation, Dereverberation And Recognition (2022)6.77
• Robust Audio-visual Target Speaker Extraction With Emotion-aware Multiple Enrollment Fusion (2025)0.00
• End-to-end Multi-talker Audio-visual ASR Using An Active Speaker Attention Module (2022)0.00