Feature Normalization For Fine-tuning Self-supervised Models In Speech Enhancement

Abstract

Large, pre-trained representation models trained using self-supervised learning have gained popularity in various fields of machine learning because they are able to extract high-quality salient features from input data. As such, they have been frequently used as base networks for various pattern classification tasks such as speech recognition. However, not much research has been conducted on applying these types of models to the field of speech signal generation. In this paper, we investigate the feasibility of using pre-trained speech representation models for a downstream speech enhancement task. To alleviate mismatches between the input features of the pre-trained model and the target enhancement model, we adopt a novel feature normalization technique to smoothly link these modules together. Our proposed method enables significant improvements in speech quality compared to baselines when combined with various types of pre-trained speech models.

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keyyang2023feature

Related papers

• Efficient Personalized Speech Enhancement Through Self-supervised Learning (2021)10.21
• Automatic Data Augmentation For Domain Adapted Fine-tuning Of Self-supervised Speech Representations (2023)0.00
• Less Forgetting For Better Generalization: Exploring Continual-learning Fine-tuning Methods For Speech Self-supervised Representations (2024)0.00
• Automatic Data Augmentation Selection And Parametrization In Contrastive Self-supervised Speech Representation Learning (2022)5.24
• Diffnorm: Self-supervised Normalization For Non-autoregressive Speech-to-speech Translation (2024)0.00
• Self-supervised Rewiring Of Pre-trained Speech Encoders: Towards Faster Fine-tuning With Less Labels In Speech Processing (2022)3.58
• Personalized Speech Enhancement Through Self-supervised Data Augmentation And Purification (2021)9.92
• Resource-efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion (2022)0.00