Effmulti: Efficiently Modeling Complex Multimodal Interactions For Emotion Analysis

Abstract

Humans are skilled in reading the interlocutor's emotion from multimodal signals, including spoken words, simultaneous speech, and facial expressions. It is still a challenge to effectively decode emotions from the complex interactions of multimodal signals. In this paper, we design three kinds of multimodal latent representations to refine the emotion analysis process and capture complex multimodal interactions from different views, including a intact three-modal integrating representation, a modality-shared representation, and three modality-individual representations. Then, a modality-semantic hierarchical fusion is proposed to reasonably incorporate these representations into a comprehensive interaction representation. The experimental results demonstrate that our EffMulti outperforms the state-of-the-art methods. The compelling performance benefits from its well-designed framework with ease of implementation, lower computing complexity, and less trainable parameters.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyqiu2022effmulti

Related papers

• Agent-based Modular Learning For Multimodal Emotion Recognition In Human-agent Systems (2025)0.00
• MIAR: Modality Interaction And Alignment Representation Fuison For Multimodal Emotion (2026)0.00
• Interpretable Multimodal Emotion Recognition Using Hybrid Fusion Of Speech And Image Data (2022)11.85
• Multimodal Emotion Recognition And Sentiment Analysis In Multi-party Conversation Contexts (2025)0.00
• Is Cross-attention Preferable To Self-attention For Multi-modal Emotion Recognition? (2022)3.64
• Multi-modal Emotion Recognition By Text, Speech And Video Using Pretrained Transformers (2024)0.00
• MMER: Multimodal Multi-task Learning For Speech Emotion Recognition (2022)10.07
• Semantic Matters: Multimodal Features For Affective Analysis (2025)0.00