Emotion Recognition By Fusing Time Synchronous And Time Asynchronous Representations
2020 · Wen Wu, Chao Zhang, Philip C. Woodland
Abstract
In this paper, a novel two-branch neural network model structure is proposed for multimodal emotion recognition, which consists of a time synchronous branch (TSB) and a time asynchronous branch (TAB). To capture correlations between each word and its acoustic realisation, the TSB combines speech and text modalities at each input window frame and then does pooling across time to form a single embedding vector. The TAB, by contrast, provides cross-utterance information by integrating sentence text embeddings from a number of context utterances into another embedding vector. The final emotion classification uses both the TSB and the TAB embeddings. Experimental results on the IEMOCAP dataset demonstrate that the two-branch structure achieves state-of-the-art results in 4-way classification with all common test setups. When using automatic speech recognition (ASR) output instead of manually transcribed reference text, it is shown that the cross-utterance information considerably improves t
Authors
(none)
Tags
Stats
Related papers
- Emotech: A Multi-modal Speech Emotion Recognition Using Multi-source Low-level Information With Hybrid Recurrent Network (2025)8.35
- Fusion Approaches For Emotion Recognition From Speech Using Acoustic And Text-based Features (2024)12.25
- Multimodal Emotion Recognition Using Transfer Learning From Speaker Recognition And Bert-based Models (2022)12.10
- Group Gated Fusion On Attention-based Bidirectional Alignment For Multimodal Emotion Recognition (2022)11.39
- A Transfer Learning Method For Speech Emotion Recognition From Automatic Speech Recognition (2020)0.00
- Multimodal Fusion With Deep Neural Networks For Audio-video Emotion Recognition (2019)0.00
- Speech Emotion Recognition Using Multi-hop Attention Mechanism (2019)14.58
- TAGF: Time-aware Gated Fusion For Multimodal Valence-arousal Estimation (2025)0.00