The Efficacy Of Self-supervised Speech Models For Audio Representations

Abstract

Self-supervised learning (SSL) speech models, which can serve as powerful upstream models to extract meaningful speech representations, have achieved unprecedented success in speech representation learning. However, their effectiveness on non-speech datasets is relatively less explored. In this work, we propose an ensemble framework, with a combination of ensemble techniques, to fuse SSL speech models' embeddings. Extensive experiments on speech and non-speech audio datasets are conducted to investigate the representation abilities of our ensemble method and its single constituent model. Ablation studies are carried out to evaluate the performances of different ensemble techniques, such as feature averaging and concatenation. All experiments are conducted during NeurIPS 2021 HEAR Challenge as a standard evaluation pipeline provided by competition officials. Results demonstrate SSL speech models' strong abilities on various non-speech tasks, while we also note that they fail to deal wit

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywu2022the

Related papers

• Investigation Of Ensemble Features Of Self-supervised Pretrained Models For Automatic Speech Recognition (2022)9.41
• Exploring Effective Fusion Algorithms For Speech Based Self-supervised Learning Models (2022)0.00
• Improving Self-supervised Learning For Audio Representations By Feature Diversity And Decorrelation (2023)0.00
• Investigating Self-supervised Learning For Speech Enhancement And Separation (2022)13.44
• Non-contrastive Self-supervised Learning For Utterance-level Information Extraction From Speech (2022)9.59
• Feature Learning And Ensemble Pre-tasks Based Self-supervised Speech Denoising And Dereverberation (2022)0.00
• Efficient Infusion Of Self-supervised Representations In Automatic Speech Recognition (2024)0.00
• Target Speech Extraction With Pre-trained Self-supervised Learning Models (2024)9.41