Face Landmark-based Speaker-independent Audio-visual Speech Enhancement In Multi-talker Environments

Abstract

In this paper, we address the problem of enhancing the speech of a speaker of interest in a cocktail party scenario when visual information of the speaker of interest is available. Contrary to most previous studies, we do not learn visual features on the typically small audio-visual datasets, but use an already available face landmark detector (trained on a separate image dataset). The landmarks are used by LSTM-based models to generate time-frequency masks which are applied to the acoustic mixed-speech spectrogram. Results show that: (i) landmark motion features are very effective features for this task, (ii) similarly to previous work, reconstruction of the target speaker's spectrogram mediated by masking is significantly more accurate than direct spectrogram reconstruction, and (iii) the best masks depend on both motion landmark features and the input mixed-speech spectrogram. To the best of our knowledge, our proposed models are the first models trained and evaluated on the limited

Stats

• citations44
• S2 citations—
• github stars0
• HF likes0
• heat score12.40
• arxiv keymorrone2018face

Related papers

• Audio-visual Target Speaker Enhancement On Multi-talker Environment Using Event-driven Cameras (2019)8.09
• Speaker-targeted Audio-visual Models For Speech Recognition In Cocktail-party Environments (2019)0.00
• An Analysis Of Speech Enhancement And Recognition Losses In Limited Resources Multi-talker Single Channel Audio-visual ASR (2019)4.52
• Lipformer: Learning To Lipread Unseen Speakers Based On Visual-landmark Transformers (2023)11.49
• Robust Audio-visual Target Speaker Extraction With Emotion-aware Multiple Enrollment Fusion (2025)0.00
• Audio2face: Generating Speech/face Animation From Single Audio With Attention-based Bidirectional LSTM Networks (2019)12.10
• Facetron: A Multi-speaker Face-to-speech Model Based On Cross-modal Latent Representations (2021)0.00
• End-to-end Multi-talker Audio-visual ASR Using An Active Speaker Attention Module (2022)0.00