Emotion2vec: Self-supervised Pre-training For Speech Emotion Representation

Abstract

We propose emotion2vec, a universal speech emotion representation model. emotion2vec is pre-trained on open-source unlabeled emotion data through self-supervised online distillation, combining utterance-level loss and frame-level loss during pre-training. emotion2vec outperforms state-of-the-art pre-trained universal models and emotion specialist models by only training linear layers for the speech emotion recognition task on the mainstream IEMOCAP dataset. In addition, emotion2vec shows consistent improvements among 10 different languages of speech emotion recognition datasets. emotion2vec also shows excellent results on other emotion tasks, such as song emotion recognition, emotion prediction in conversation, and sentiment analysis. Comparison experiments, ablation experiments, and visualization comprehensively demonstrate the universal capability of the proposed emotion2vec. To the best of our knowledge, emotion2vec is the first universal representation model in various emotion-rela

Stats

• citations130
• S2 citations—
• github stars0
• HF likes0
• heat score15.88
• arxiv keyma2023emotion2vec

Related papers

• Unsupervised Representations Improve Supervised Learning In Speech Emotion Recognition (2023)0.00
• Pre-trained Model Representations And Their Robustness Against Noise For Speech Emotion Analysis (2023)0.00
• Exploring Wav2vec 2.0 Fine-tuning For Improved Speech Emotion Recognition (2021)15.67
• Representation Learning Through Cross-modal Conditional Teacher-student Training For Speech Emotion Recognition (2021)11.19
• Speaker Emotion Recognition: Leveraging Self-supervised Models For Feature Extraction Using Wav2vec2 And Hubert (2024)0.00
• On The Use Of Self-supervised Pre-trained Acoustic And Linguistic Features For Continuous Speech Emotion Recognition (2020)11.85
• Attention Based Fully Convolutional Network For Speech Emotion Recognition (2018)15.25
• Data2vec: A General Framework For Self-supervised Learning In Speech, Vision And Language (2022)0.00