Audio-visual Compound Expression Recognition Method Based On Late Modality Fusion And Rule-based Decision

Abstract

This paper presents the results of the SUN team for the Compound Expressions Recognition Challenge of the 6th ABAW Competition. We propose a novel audio-visual method for compound expression recognition. Our method relies on emotion recognition models that fuse modalities at the emotion probability level, while decisions regarding the prediction of compound expressions are based on predefined rules. Notably, our method does not use any training data specific to the target task. Thus, the problem is a zero-shot classification task. The method is evaluated in multi-corpus training and cross-corpus validation setups. Using our proposed method is achieved an F1-score value equals to 22.01% on the C-EXPR-DB test subset. Our findings from the challenge demonstrate that the proposed method can potentially form a basis for developing intelligent tools for annotating audio-visual data in the context of human's basic and compound emotions.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyryumina2024audio

Related papers

• SUN Team's Contribution To ABAW 2024 Competition: Audio-visual Valence-arousal Estimation And Expression Recognition (2024)0.00
• Multimodal Fusion Method With Spatiotemporal Sequences And Relationship Learning For Valence-arousal Estimation (2024)0.00
• Temporal Aggregation Of Audio-visual Modalities For Emotion Recognition (2020)8.09
• Continuous Multimodal Emotion Recognition Approach For AVEC 2017 (2017)0.00
• A Joint Cross-attention Model For Audio-visual Fusion In Dimensional Emotion Recognition (2022)18.00
• Recursive Joint Attention For Audio-visual Fusion In Regression Based Emotion Recognition (2023)9.59
• Emotion Recognition System From Speech And Visual Information Based On Convolutional Neural Networks (2020)10.21
• Mutilmodal Feature Extraction And Attention-based Fusion For Emotion Estimation In Videos (2023)1.40