Improving Speech Emotion Recognition In Under-resourced Languages Via Speech-to-speech Translation With Bootstrapping Data Selection

Abstract

Speech Emotion Recognition (SER) is a crucial component in developing general-purpose AI agents capable of natural human-computer interaction. However, building robust multilingual SER systems remains challenging due to the scarcity of labeled data in languages other than English and Chinese. In this paper, we propose an approach to enhance SER performance in low SER resource languages by leveraging data from high-resource languages. Specifically, we employ expressive Speech-to-Speech translation (S2ST) combined with a novel bootstrapping data selection pipeline to generate labeled data in the target language. Extensive experiments demonstrate that our method is both effective and generalizable across different upstream models and languages. Our results suggest that this approach can facilitate the development of more scalable and robust multilingual SER systems.

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keylin2024improving

Related papers

• Leveraging Unsupervised And Weakly-supervised Data To Improve Direct Speech-to-speech Translation (2022)8.35
• Improved Speech Emotion Recognition Using Transfer Learning And Spectrogram Augmentation (2021)12.74
• Semi-supervised Cross-lingual Speech Emotion Recognition (2022)10.85
• Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, And Augmenting (2023)0.00
• Continuous Metric Learning For Transferable Speech Emotion Recognition And Embedding Across Low-resource Languages (2022)0.00
• Leveraging Pseudo-labeled Data To Improve Direct Speech-to-speech Translation (2022)10.33
• Enhanced Direct Speech-to-speech Translation Using Self-supervised Pre-training And Data Augmentation (2022)10.85
• Generative Data Augmentation Guided By Triplet Loss For Speech Emotion Recognition (2022)3.58