Muse: Multi-modal Target Speaker Extraction With Visual Cues

Abstract

Speaker extraction algorithm relies on the speech sample from the target speaker as the reference point to focus its attention. Such a reference speech is typically pre-recorded. On the other hand, the temporal synchronization between speech and lip movement also serves as an informative cue. Motivated by this idea, we study a novel technique to use speech-lip visual cues to extract reference target speech directly from mixture speech during inference time, without the need of pre-recorded reference speech. We propose a multi-modal speaker extraction network, named MuSE, that is conditioned only on a lip image sequence. MuSE not only outperforms other competitive baselines in terms of SI-SDR and PESQ, but also shows consistent improvement in cross-dataset evaluations.

Stats

• citations37
• S2 citations—
• github stars0
• HF likes0
• heat score11.85
• arxiv keypan2020muse

Related papers

• USEV: Universal Speaker Extraction With Visual Cue (2021)12.17
• Selective Listening By Synchronizing Speech With Lips (2021)11.85
• Limuse: Lightweight Multi-modal Speaker Extraction (2021)0.00
• Imaginenet: Target Speaker Extraction With Intermittent Visual Cue Through Embedding Inpainting (2022)7.16
• New Insights On Target Speaker Extraction (2022)0.00
• A Single Speech Enhancement Model Unifying Dereverberation, Denoising, Speaker Counting, Separation, And Extraction (2023)7.16
• Robust Audio-visual Target Speaker Extraction With Emotion-aware Multiple Enrollment Fusion (2025)0.00
• Enhancing Real-world Active Speaker Detection With Multi-modal Extraction Pre-training (2024)5.24