Universal Paralinguistic Speech Representations Using Self-supervised Conformers

Abstract

Many speech applications require understanding aspects beyond the words being spoken, such as recognizing emotion, detecting whether the speaker is wearing a mask, or distinguishing real from synthetic speech. In this work, we introduce a new state-of-the-art paralinguistic representation derived from large-scale, fully self-supervised training of a 600M+ parameter Conformer-based architecture. We benchmark on a diverse set of speech tasks and demonstrate that simple linear classifiers trained on top of our time-averaged representation outperform nearly all previous results, in some cases by large margins. Our analyses of context-window size demonstrate that, surprisingly, 2 second context-windows achieve 96% the performance of the Conformers that use the full long-term context on 7 out of 9 tasks. Furthermore, while the best per-task representations are extracted internally in the network, stable performance across several layers allows a single universal representation to reach near

Stats

• citations24
• S2 citations—
• github stars0
• HF likes0
• heat score10.48
• arxiv keyshor2021universal

Related papers

• Conformer-based Self-supervised Learning For Non-speech Audio Tasks (2021)7.50
• Learning Problem-agnostic Speech Representations From Multiple Self-supervised Tasks (2019)15.54
• Speech Representation Analysis Based On Inter- And Intra-model Similarities (2024)2.26
• Fast Conformer With Linearly Scalable Attention For Efficient Speech Recognition (2023)14.47
• Unispeech-sat: Universal Speech Representation Learning With Speaker Aware Pre-training (2021)0.00
• Accidental Learners: Spoken Language Identification In Multilingual Self-supervised Models (2022)5.84
• PCNN: A Lightweight Parallel Conformer Neural Network For Efficient Monaural Speech Enhancement (2023)6.77
• Automatic Data Augmentation Selection And Parametrization In Contrastive Self-supervised Speech Representation Learning (2022)5.24