Limuse: Lightweight Multi-modal Speaker Extraction

Abstract

Multi-modal cues, including spatial information, facial expression and voiceprint, are introduced to the speech separation and speaker extraction tasks to serve as complementary information to achieve better performance. However, the introduction of these cues brings about an increasing number of parameters and model complexity, which makes it harder to deploy these models on resource-constrained devices. In this paper, we alleviate the aforementioned problem by proposing a Lightweight Multi-modal framework for Speaker Extraction (LiMuSE). We propose to use GC-equipped TCN, which incorporates Group Communication (GC) and Temporal Convolutional Network (TCN) in the Context Codec module, the audio block and the fusion block. The experiments on the MC_GRID dataset demonstrate that LiMuSE achieves on par or better performance with a much smaller number of parameters and less model complexity. We further investigate the impacts of the quantization of LiMuSE. Our code and dataset are provide

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2021limuse

Related papers

• Muse: Multi-modal Target Speaker Extraction With Visual Cues (2020)11.85
• Mc-spex: Towards Effective Speaker Extraction With Multi-scale Interfusion And Conditional Speaker Modulation (2023)9.23
• Triplec Learning And Lightweight Speech Enhancement For Multi-condition Target Speech Extraction (2025)0.00
• Lightweight Speech Enhancement Guided Target Speech Extraction In Noisy Multi-speaker Scenarios (2025)0.00
• 3S-TSE: Efficient Three-stage Target Speaker Extraction For Real-time And Low-resource Applications (2023)5.24
• Lisennet: Lightweight Sub-band And Dual-path Modeling For Real-time Speech Enhancement (2024)9.03
• Typing To Listen At The Cocktail Party: Text-guided Target Speaker Extraction (2023)3.58
• Single Microphone Speaker Extraction Using Unified Time-frequency Siamese-unet (2022)3.58