Multi-modal Emotion Detection With Transfer Learning

Abstract

Automated emotion detection in speech is a challenging task due to the complex interdependence between words and the manner in which they are spoken. It is made more difficult by the available datasets; their small size and incompatible labeling idiosyncrasies make it hard to build generalizable emotion detection systems. To address these two challenges, we present a multi-modal approach that first transfers learning from related tasks in speech and text to produce robust neural embeddings and then uses these embeddings to train a pLDA classifier that is able to adapt to previously unseen emotions and domains. We begin by training a multilayer TDNN on the task of speaker identification with the VoxCeleb corpora and then fine-tune it on the task of emotion identification with the Crema-D corpus. Using this network, we extract speech embeddings for Crema-D from each of its layers, generate and concatenate text embeddings for the accompanying transcripts using a fine-tuned BERT model and

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyananthram2020multi

Related papers

• Multimodal Emotion Recognition Using Transfer Learning From Speaker Recognition And Bert-based Models (2022)12.10
• A Transfer Learning Method For Speech Emotion Recognition From Automatic Speech Recognition (2020)0.00
• Transfer Learning For Improving Speech Emotion Classification Accuracy (2018)15.10
• Embedded Emotions -- A Data Driven Approach To Learn Transferable Feature Representations From Raw Speech Input For Emotion Recognition (2020)0.00
• Emodiarize: Speaker Diarization And Emotion Identification From Speech Signals Using Convolutional Neural Networks (2023)0.00
• Progressive Neural Networks For Transfer Learning In Emotion Recognition (2017)14.19
• Emonet: A Transfer Learning Framework For Multi-corpus Speech Emotion Recognition (2021)2.95
• Continuous Metric Learning For Transferable Speech Emotion Recognition And Embedding Across Low-resource Languages (2022)0.00