Multimodal Fusion With Deep Neural Networks For Audio-video Emotion Recognition

Abstract

This paper presents a novel deep neural network (DNN) for multimodal fusion of audio, video and text modalities for emotion recognition. The proposed DNN architecture has independent and shared layers which aim to learn the representation for each modality, as well as the best combined representation to achieve the best prediction. Experimental results on the AVEC Sentiment Analysis in the Wild dataset indicate that the proposed DNN can achieve a higher level of Concordance Correlation Coefficient (CCC) than other state-of-the-art systems that perform early fusion of modalities at feature-level (i.e., concatenation) and late fusion at score-level (i.e., weighted average) fusion. The proposed DNN has achieved CCCs of 0.606, 0.534, and 0.170 on the development partition of the dataset for predicting arousal, valence and liking, respectively.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyortega2019multimodal

Related papers

• Multistage Linguistic Conditioning Of Convolutional Layers For Speech Emotion Recognition (2021)9.23
• Multimodal Speech Emotion Recognition Using Audio And Text (2018)18.02
• A Joint Cross-attention Model For Audio-visual Fusion In Dimensional Emotion Recognition (2022)18.00
• Continuous Multimodal Emotion Recognition Approach For AVEC 2017 (2017)0.00
• Emotion Recognition System From Speech And Visual Information Based On Convolutional Neural Networks (2020)10.21
• Interpretable Multimodal Emotion Recognition Using Hybrid Fusion Of Speech And Image Data (2022)11.85
• Temporal Aggregation Of Audio-visual Modalities For Emotion Recognition (2020)8.09
• A Multimodal Approach Towards Emotion Recognition Of Music Using Audio And Lyrical Content (2018)0.00