Jointly Aligning And Predicting Continuous Emotion Annotations

Abstract

Time-continuous dimensional descriptions of emotions (e.g., arousal, valence) allow researchers to characterize short-time changes and to capture long-term trends in emotion expression. However, continuous emotion labels are generally not synchronized with the input speech signal due to delays caused by reaction-time, which is inherent in human evaluations. To deal with this challenge, we introduce a new convolutional neural network (multi-delay sinc network) that is able to simultaneously align and predict labels in an end-to-end manner. The proposed network is a stack of convolutional layers followed by an aligner network that aligns the speech signal and emotion labels. This network is implemented using a new convolutional layer that we introduce, the delayed sinc layer. It is a time-shifted low-pass (sinc) filter that uses a gradient-based algorithm to learn a single delay. Multiple delayed sinc layers can be used to compensate for a non-stationary delay that is a function of the a

Stats

• citations13
• S2 citations—
• github stars0
• HF likes0
• heat score8.60
• arxiv keykhorram2019jointly

Related papers

• Dynamic Time-alignment Of Dimensional Annotations Of Emotion Using Recurrent Neural Networks (2022)0.00
• Capturing Long-term Temporal Dependencies With Convolutional Networks For Continuous Emotion Recognition (2017)10.48
• Unifying The Discrete And Continuous Emotion Labels For Speech Emotion Recognition (2022)0.00
• Attention Based Fully Convolutional Network For Speech Emotion Recognition (2018)15.25
• Learning Alignment For Multimodal Emotion Recognition From Speech (2019)15.22
• Mingling Or Misalignment? Temporal Shift For Speech Emotion Recognition With Pre-trained Representations (2023)13.84
• Emotion Recognition By Fusing Time Synchronous And Time Asynchronous Representations (2020)13.05
• Multistage Linguistic Conditioning Of Convolutional Layers For Speech Emotion Recognition (2021)9.23