Double Multi-head Attention Multimodal System For Odyssey 2024 Speech Emotion Recognition Challenge

Abstract

As computer-based applications are becoming more integrated into our daily lives, the importance of Speech Emotion Recognition (SER) has increased significantly. Promoting research with innovative approaches in SER, the Odyssey 2024 Speech Emotion Recognition Challenge was organized as part of the Odyssey 2024 Speaker and Language Recognition Workshop. In this paper we describe the Double Multi-Head Attention Multimodal System developed for this challenge. Pre-trained self-supervised models were used to extract informative acoustic and text features. An early fusion strategy was adopted, where a Multi-Head Attention layer transforms these mixed features into complementary contextualized representations. A second attention mechanism is then applied to pool these representations into an utterance-level vector. Our proposed system achieved the third position in the categorical task ranking with a 34.41% Macro-F1 score, where 31 teams participated in total.

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keycosta2024double

Related papers

• Cross-language Speech Emotion Recognition Using Multimodal Dual Attention Transformers (2023)0.00
• Speech Emotion Recognition Using Multi-hop Attention Mechanism (2019)14.58
• Msp-podcast SER Challenge 2024: L'antenne Du Ventoux Multimodal Self-supervised Learning For Speech Emotion Recognition (2024)5.84
• MMER: Multimodal Multi-task Learning For Speech Emotion Recognition (2022)10.07
• Speech Emotion Recognition With Co-attention Based Multi-level Acoustic Information (2022)16.17
• Exploring Attention Mechanisms For Multimodal Emotion Recognition In An Emergency Call Center Corpus (2023)8.09
• Multistage Linguistic Conditioning Of Convolutional Layers For Speech Emotion Recognition (2021)9.23
• Multimodal Emotion Recognition And Sentiment Analysis In Multi-party Conversation Contexts (2025)0.00