Exploring Self-supervised Multi-view Contrastive Learning For Speech Emotion Recognition With Limited Annotations

Abstract

Recent advancements in Deep and Self-Supervised Learning (SSL) have led to substantial improvements in Speech Emotion Recognition (SER) performance, reaching unprecedented levels. However, obtaining sufficient amounts of accurately labeled data for training or fine-tuning the models remains a costly and challenging task. In this paper, we propose a multi-view SSL pre-training technique that can be applied to various representations of speech, including the ones generated by large speech models, to improve SER performance in scenarios where annotations are limited. Our experiments, based on wav2vec 2.0, spectral and paralinguistic features, demonstrate that the proposed framework boosts the SER performance, by up to 10% in Unweighted Average Recall, in settings with extremely sparse data annotations.

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keykhaertdinov2024exploring

Related papers

• Semi-supervised Cross-lingual Speech Emotion Recognition (2022)10.85
• Leveraging Semantic Information For Efficient Self-supervised Emotion Recognition With Audio-textual Distilled Models (2023)6.34
• End-to-end Integration Of Speech Emotion Recognition With Voice Activity Detection Using Self-supervised Learning Features (2024)0.00
• Cross-lingual Speech Emotion Recognition: Humans Vs. Self-supervised Models (2024)5.84
• Towards Supervised Performance On Speaker Verification With Self-supervised Learning By Leveraging Large-scale ASR Models (2024)7.50
• Supervised Contrastive Learning With Nearest Neighbor Search For Speech Emotion Recognition (2023)7.16
• SER Evals: In-domain And Out-of-domain Benchmarking For Speech Emotion Recognition (2024)4.52
• Non-contrastive Self-supervised Learning For Utterance-level Information Extraction From Speech (2022)9.59