Investigating Self-supervised Learning For Speech Enhancement And Separation

Abstract

Speech enhancement and separation are two fundamental tasks for robust speech processing. Speech enhancement suppresses background noise while speech separation extracts target speech from interfering speakers. Despite a great number of supervised learning-based enhancement and separation methods having been proposed and achieving good performance, studies on applying self-supervised learning (SSL) to enhancement and separation are limited. In this paper, we evaluate 13 SSL upstream methods on speech enhancement and separation downstream tasks. Our experimental results on Voicebank-DEMAND and Libri2Mix show that some SSL representations consistently outperform baseline features including the short-time Fourier transform (STFT) magnitude and log Mel filterbank (FBANK). Furthermore, we analyze the factors that make existing SSL frameworks difficult to apply to speech enhancement and separation and discuss the representation properties desired for both tasks. Our study is included as the

Stats

• citations61
• S2 citations—
• github stars0
• HF likes0
• heat score13.44
• arxiv keyhuang2022investigating

Related papers

• Downstream Task Agnostic Speech Enhancement With Self-supervised Representation Loss (2023)6.77
• Exploring Effective Fusion Algorithms For Speech Based Self-supervised Learning Models (2022)0.00
• Feature Learning And Ensemble Pre-tasks Based Self-supervised Speech Denoising And Dereverberation (2022)0.00
• Analyzing The Factors Affecting Usefulness Of Self-supervised Pre-trained Representations For Speech Recognition (2022)0.00
• Deploying Self-supervised Learning In The Wild For Hybrid Automatic Speech Recognition (2022)0.00
• The Efficacy Of Self-supervised Speech Models For Audio Representations (2022)0.00
• Lebenchmark: A Reproducible Framework For Assessing Self-supervised Representation Learning From Speech (2021)11.39
• Target Speech Extraction With Pre-trained Self-supervised Learning Models (2024)9.41