Mingling Or Misalignment? Temporal Shift For Speech Emotion Recognition With Pre-trained Representations

Abstract

Fueled by recent advances of self-supervised models, pre-trained speech representations proved effective for the downstream speech emotion recognition (SER) task. Most prior works mainly focus on exploiting pre-trained representations and just adopt a linear head on top of the pre-trained model, neglecting the design of the downstream network. In this paper, we propose a temporal shift module to mingle channel-wise information without introducing any parameter or FLOP. With the temporal shift module, three designed baseline building blocks evolve into corresponding shift variants, i.e. ShiftCNN, ShiftLSTM, and Shiftformer. Moreover, to balance the trade-off between mingling and misalignment, we propose two technical strategies, placement of shift and proportion of shift. The family of temporal shift models all outperforms the state-of-the-art methods on the benchmark IEMOCAP dataset under both finetuning and feature extraction settings. Our code is available at https://github.com/ECNU-

Stats

• citations25
• S2 citations—
• github stars40
• HF likes0
• heat score13.84
• arxiv keyshen2023mingling

Related papers

• Unsupervised Representations Improve Supervised Learning In Speech Emotion Recognition (2023)0.00
• Jointly Fine-tuning "bert-like" Self Supervised Models To Improve Multimodal Speech Emotion Recognition (2020)13.74
• Pre-trained Model Representations And Their Robustness Against Noise For Speech Emotion Analysis (2023)0.00
• Multistage Linguistic Conditioning Of Convolutional Layers For Speech Emotion Recognition (2021)9.23
• CTA-RNN: Channel And Temporal-wise Attention RNN Leveraging Pre-trained ASR Embeddings For Speech Emotion Recognition (2022)5.84
• Group Gated Fusion On Attention-based Bidirectional Alignment For Multimodal Emotion Recognition (2022)11.39
• Emotion Recognition By Fusing Time Synchronous And Time Asynchronous Representations (2020)13.05
• Leveraging Speech PTM, Text LLM, And Emotional TTS For Speech Emotion Recognition (2023)10.97