Unifying Speech Enhancement And Separation With Gradient Modulation For End-to-end Noise-robust Speech Separation

Abstract

Recent studies in neural network-based monaural speech separation (SS) have achieved a remarkable success thanks to increasing ability of long sequence modeling. However, they would degrade significantly when put under realistic noisy conditions, as the background noise could be mistaken for speaker's speech and thus interfere with the separated sources. To alleviate this problem, we propose a novel network to unify speech enhancement and separation with gradient modulation to improve noise-robustness. Specifically, we first build a unified network by combining speech enhancement (SE) and separation modules, with multi-task learning for optimization, where SE is supervised by parallel clean mixture to reduce noise for downstream speech separation. Furthermore, in order to avoid suppressing valid speaker information when reducing noise, we propose a gradient modulation (GM) strategy to harmonize the SE and SS tasks from optimization view. Experimental results show that our approach achi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhu2023unifying

Related papers

• End-to-end Networks For Supervised Single-channel Speech Separation (2018)0.00
• Audio-visual Speech Separation And Dereverberation With A Two-stage Multimodal Network (2019)12.47
• Two-stage Model And Optimal SI-SNR For Monaural Multi-speaker Speech Separation In Noisy Environment (2020)0.00
• A Multi-stage Triple-path Method For Speech Separation In Noisy And Reverberant Environments (2023)2.26
• Noise-aware Speech Separation With Contrastive Learning (2023)6.77
• Bridging The Gap: Integrating Pre-trained Speech Enhancement And Recognition Models For Robust Speech Recognition (2024)7.50
• Real-time Speech Enhancement And Separation With A Unified Deep Neural Network For Single/dual Talker Scenarios (2023)2.26
• GSEP: A Robust Vocal And Accompaniment Separation System Using Gated CBHG Module And Loudness Normalization (2020)0.00