Selective Listening By Synchronizing Speech With Lips

Abstract

A speaker extraction algorithm seeks to extract the speech of a target speaker from a multi-talker speech mixture when given a cue that represents the target speaker, such as a pre-enrolled speech utterance, or an accompanying video track. Visual cues are particularly useful when a pre-enrolled speech is not available. In this work, we don't rely on the target speaker's pre-enrolled speech, but rather use the target speaker's face track as the speaker cue, that is referred to as the auxiliary reference, to form an attractor towards the target speaker. We advocate that the temporal synchronization between the speech and its accompanying lip movements is a direct and dominant audio-visual cue. Therefore, we propose a self-supervised pre-training strategy, to exploit the speech-lip synchronization cue for target speaker extraction, which allows us to leverage abundant unlabeled in-domain data. We transfer the knowledge from the pre-trained model to the attractor encoder of the speaker ext

Stats

• citations37
• S2 citations—
• github stars0
• HF likes0
• heat score11.85
• arxiv keypan2021selective

Related papers

• Muse: Multi-modal Target Speaker Extraction With Visual Cues (2020)11.85
• USEV: Universal Speaker Extraction With Visual Cue (2021)12.17
• Audio-visual Active Speaker Extraction For Sparsely Overlapped Multi-talker Speech (2023)7.50
• Target Speaker Lipreading By Audio-visual Self-distillation Pretraining And Speaker Adaptation (2025)5.24
• Imaginenet: Target Speaker Extraction With Intermittent Visual Cue Through Embedding Inpainting (2022)7.16
• Multi-grained Spatio-temporal Modeling For Lip-reading (2019)0.00
• Lipsound2: Self-supervised Pre-training For Lip-to-speech Reconstruction And Lip Reading (2021)11.39
• Typing To Listen At The Cocktail Party: Text-guided Target Speaker Extraction (2023)3.58