Is Cross-attention Preferable To Self-attention For Multi-modal Emotion Recognition?

Abstract

Humans express their emotions via facial expressions, voice intonation and word choices. To infer the nature of the underlying emotion, recognition models may use a single modality, such as vision, audio, and text, or a combination of modalities. Generally, models that fuse complementary information from multiple modalities outperform their uni-modal counterparts. However, a successful model that fuses modalities requires components that can effectively aggregate task-relevant information from each modality. As cross-modal attention is seen as an effective mechanism for multi-modal fusion, in this paper we quantify the gain that such a mechanism brings compared to the corresponding self-attention mechanism. To this end, we implement and compare a cross-attention and a self-attention model. In addition to attention, each model uses convolutional layers for local feature extraction and recurrent layers for global sequential modelling. We compare the models using different modality combin

Stats

• citations0
• S2 citations—
• github stars65
• HF likes0
• heat score3.64
• arxiv keyrajan2022is

Related papers

• A Joint Cross-attention Model For Audio-visual Fusion In Dimensional Emotion Recognition (2022)18.00
• Exploring Attention Mechanisms For Multimodal Emotion Recognition In An Emergency Call Center Corpus (2023)8.09
• Cross-modal Fusion Techniques For Utterance-level Emotion Recognition From Text And Speech (2023)9.59
• Multimodal Speech Emotion Recognition Using Cross Attention With Aligned Audio And Text (2022)9.76
• Effmulti: Efficiently Modeling Complex Multimodal Interactions For Emotion Analysis (2022)0.00
• Attentive Modality Hopping Mechanism For Speech Emotion Recognition (2019)0.00
• Speech Emotion Recognition Using Multi-hop Attention Mechanism (2019)14.58
• Learning Alignment For Multimodal Emotion Recognition From Speech (2019)15.22