Pre-trained Model Representations And Their Robustness Against Noise For Speech Emotion Analysis

Abstract

Pre-trained model representations have demonstrated state-of-the-art performance in speech recognition, natural language processing, and other applications. Speech models, such as Bidirectional Encoder Representations from Transformers (BERT) and Hidden units BERT (HuBERT), have enabled generating lexical and acoustic representations to benefit speech recognition applications. We investigated the use of pre-trained model representations for estimating dimensional emotions, such as activation, valence, and dominance, from speech. We observed that while valence may rely heavily on lexical representations, activation and dominance rely mostly on acoustic information. In this work, we used multi-modal fusion representations from pre-trained models to generate state-of-the-art speech emotion estimation, and we showed a 100% and 30% relative improvement in concordance correlation coefficient (CCC) on valence estimation compared to standard acoustic and lexical baselines. Finally, we investig

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keymitra2023pre

Related papers

• Speech Emotion: Investigating Model Representations, Multi-task Learning And Knowledge Distillation (2022)6.34
• Investigating Salient Representations And Label Variance In Dimensional Speech Emotion Analysis (2023)3.58
• An Analysis Of Large Speech Models-based Representations For Speech Emotion Recognition (2023)4.52
• Representation Learning Through Cross-modal Conditional Teacher-student Training For Speech Emotion Recognition (2021)11.19
• Decoding Emotions: A Comprehensive Multilingual Study Of Speech Models For Speech Emotion Recognition (2023)0.00
• Dawn Of The Transformer Era In Speech Emotion Recognition: Closing The Valence Gap (2022)18.59
• Advancing Audio Emotion And Intent Recognition With Large Pre-trained Models And Bayesian Inference (2023)5.24
• Jointly Fine-tuning "bert-like" Self Supervised Models To Improve Multimodal Speech Emotion Recognition (2020)13.74