Leveraging Semantic Information For Efficient Self-supervised Emotion Recognition With Audio-textual Distilled Models

Abstract

In large part due to their implicit semantic modeling, self-supervised learning (SSL) methods have significantly increased the performance of valence recognition in speech emotion recognition (SER) systems. Yet, their large size may often hinder practical implementations. In this work, we take HuBERT as an example of an SSL model and analyze the relevance of each of its layers for SER. We show that shallow layers are more important for arousal recognition while deeper layers are more important for valence. This observation motivates the importance of additional textual information for accurate valence recognition, as the distilled framework lacks the depth of its large-scale SSL teacher. Thus, we propose an audio-textual distilled SSL framework that, while having only ~20% of the trainable parameters of a large SSL model, achieves on par performance across the three emotion dimensions (arousal, valence, dominance) on the MSP-Podcast v1.10 dataset.

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keydeoliveira2023leveraging

Related papers

• Exploring Self-supervised Multi-view Contrastive Learning For Speech Emotion Recognition With Limited Annotations (2024)3.58
• Jointly Fine-tuning "bert-like" Self Supervised Models To Improve Multimodal Speech Emotion Recognition (2020)13.74
• Speech Emotion Recognition With Distilled Prosodic And Linguistic Affect Representations (2023)5.24
• Cross-lingual Speech Emotion Recognition: Humans Vs. Self-supervised Models (2024)5.84
• Exploring Acoustic Similarity In Emotional Speech And Music Via Self-supervised Representations (2024)3.58
• Star: Distilling Speech Temporal Relation For Lightweight Speech Self-supervised Learning Models (2023)5.24
• The Efficacy Of Self-supervised Speech Models For Audio Representations (2022)0.00
• Fithubert: Going Thinner And Deeper For Knowledge Distillation Of Speech Self-supervised Learning (2022)10.97