Speech Emotion Recognition With Co-attention Based Multi-level Acoustic Information

Abstract

Speech Emotion Recognition (SER) aims to help the machine to understand human's subjective emotion from only audio information. However, extracting and utilizing comprehensive in-depth audio information is still a challenging task. In this paper, we propose an end-to-end speech emotion recognition system using multi-level acoustic information with a newly designed co-attention module. We firstly extract multi-level acoustic information, including MFCC, spectrogram, and the embedded high-level acoustic information with CNN, BiLSTM and wav2vec2, respectively. Then these extracted features are treated as multimodal inputs and fused by the proposed co-attention mechanism. Experiments are carried on the IEMOCAP dataset, and our model achieves competitive performance with two different speaker-independent cross-validation strategies. Our code is available on GitHub.

Stats

• citations142
• S2 citations—
• github stars0
• HF likes0
• heat score16.17
• arxiv keyzou2022speech

Related papers

• Speech Emotion Recognition With Multiscale Area Attention And Data Augmentation (2021)13.65
• Attention Based Fully Convolutional Network For Speech Emotion Recognition (2018)15.25
• Speech Emotion Recognition Using Multi-hop Attention Mechanism (2019)14.58
• Speech Emotion Recognition Via Cnn-transformer And Multidimensional Attention Mechanism (2024)0.00
• Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-attention Cues In Multitask Learning (2024)0.00
• Emotech: A Multi-modal Speech Emotion Recognition Using Multi-source Low-level Information With Hybrid Recurrent Network (2025)8.35
• Leveraging Cross-attention Transformer And Multi-feature Fusion For Cross-linguistic Speech Emotion Recognition (2025)4.52
• Enhanced Speech Emotion Recognition With Efficient Channel Attention Guided Deep Cnn-bilstm Framework (2024)0.00