Pretext Tasks Selection For Multitask Self-supervised Speech Representation Learning

Abstract

Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task. In audio/speech signal processing, a wide range of features where engineered through decades of research efforts. As it turns out, learning to predict such features (a.k.a pseudo-labels) has proven to be a particularly relevant pretext task, leading to useful self-supervised representations which prove to be effective for downstream tasks. However, methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly. In fact, the process relies almost exclusively on a computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks. This paper introduces a method to select a group of pretext tasks among a set of candidates. The method we propose estimates calibrated

Stats

• citations13
• S2 citations—
• github stars0
• HF likes0
• heat score8.60
• arxiv keyzaiem2021pretext

Related papers

• Automatic Data Augmentation Selection And Parametrization In Contrastive Self-supervised Speech Representation Learning (2022)5.24
• Learning Problem-agnostic Speech Representations From Multiple Self-supervised Tasks (2019)15.54
• Self-supervised Learning Based Monaural Speech Enhancement With Multi-task Pre-training (2021)0.00
• Feature Learning And Ensemble Pre-tasks Based Self-supervised Speech Denoising And Dereverberation (2022)0.00
• Similarity Analysis Of Self-supervised Speech Representations (2020)10.07
• Progressive Residual Extraction Based Pre-training For Speech Representation Learning (2024)0.00
• Application Of Knowledge Distillation To Multi-task Speech Representation Learning (2022)2.26
• Selective Hubert: Self-supervised Pre-training For Target Speaker In Clean And Mixture Speech (2023)7.81