Wavlm: Large-scale Self-supervised Pre-training For Full Stack Speech Processing
2021 · Sanyuan Chen, Chengyi Wang, Zhengyang Chen, et al.
Abstract
Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benc
Authors
(none)
Tags
Stats
Related papers
- Attentive Merging Of Hidden Embeddings From Pre-trained Speech Model For Anti-spoofing Detection (2024)0.00
- Large Language Model Guided Decoding For Self-supervised Speech Recognition (2025)0.00
- An Adapter Based Pre-training For Efficient And Scalable Self-supervised Speech Representation Learning (2021)8.35
- Weakly-supervised Speech Pre-training: A Case Study On Target Speech Recognition (2023)8.09
- Wav2vec 2.0: A Framework For Self-supervised Learning Of Speech Representations (2020)0.00
- A Noise-robust Self-supervised Pre-training Model Based Speech Representation Learning For Automatic Speech Recognition (2022)11.19
- Multi-task Voice Activated Framework Using Self-supervised Learning (2021)6.34
- Towards Supervised Performance On Speaker Verification With Self-supervised Learning By Leveraging Large-scale ASR Models (2024)7.50