Exploiting Modality-invariant Feature For Robust Multimodal Emotion Recognition With Missing Modalities

Abstract

Multimodal emotion recognition leverages complementary information across modalities to gain performance. However, we cannot guarantee that the data of all modalities are always present in practice. In the studies to predict the missing data across modalities, the inherent difference between heterogeneous modalities, namely the modality gap, presents a challenge. To address this, we propose to use invariant features for a missing modality imagination network (IF-MMIN) which includes two novel mechanisms: 1) an invariant feature learning strategy that is based on the central moment discrepancy (CMD) distance under the full-modality scenario; 2) an invariant feature based imagination module (IF-IM) to alleviate the modality gap during the missing modalities prediction, thus improving the robustness of multimodal joint representation. Comprehensive experiments on the benchmark dataset IEMOCAP demonstrate that the proposed model outperforms all baselines and invariantly improves the overal

Stats

• citations0
• S2 citations—
• github stars37
• HF likes0
• heat score3.16
• arxiv keyzuo2022exploiting

Related papers

• MIAR: Modality Interaction And Alignment Representation Fuison For Multimodal Emotion (2026)0.00
• Effmulti: Efficiently Modeling Complex Multimodal Interactions For Emotion Analysis (2022)0.00
• Attentive Modality Hopping Mechanism For Speech Emotion Recognition (2019)0.00
• Interpretable Multimodal Emotion Recognition Using Hybrid Fusion Of Speech And Image Data (2022)11.85
• Enhancing Multimodal Sentiment Analysis For Missing Modality Through Self-distillation And Unified Modality Cross-attention (2024)6.71
• Enhancing Modal Fusion By Alignment And Label Matching For Multimodal Emotion Recognition (2024)6.34
• Contrastive Regularization For Multimodal Emotion Recognition Using Audio And Text (2022)0.00
• MMER: Multimodal Multi-task Learning For Speech Emotion Recognition (2022)10.07