Extending Rnn-t-based Speech Recognition Systems With Emotion And Language Classification

Abstract

Speech transcription, emotion recognition, and language identification are usually considered to be three different tasks. Each one requires a different model with a different architecture and training process. We propose using a recurrent neural network transducer (RNN-T)-based speech-to-text (STT) system as a common component that can be used for emotion recognition and language identification as well as for speech recognition. Our work extends the STT system for emotion classification through minimal changes, and shows successful results on the IEMOCAP and MELD datasets. In addition, we demonstrate that by adding a lightweight component to the RNN-T module, it can also be used for language identification. In our evaluations, this new classifier demonstrates state-of-the-art accuracy for the NIST-LRE-07 dataset.

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keykons2022extending

Related papers

• CTA-RNN: Channel And Temporal-wise Attention RNN Leveraging Pre-trained ASR Embeddings For Speech Emotion Recognition (2022)5.84
• A Transfer Learning Method For Speech Emotion Recognition From Automatic Speech Recognition (2020)0.00
• Emotion Recognition From Speech With Recurrent Neural Networks (2017)0.00
• Emotech: A Multi-modal Speech Emotion Recognition Using Multi-source Low-level Information With Hybrid Recurrent Network (2025)8.35
• Incorporating End-to-end Speech Recognition Models For Sentiment Analysis (2019)9.59
• Emotion Recognition From Speech (2019)0.00
• Deep Learning Based Emotion Recognition System Using Speech Features And Transcriptions (2019)0.00
• Multimodal Speech Emotion Recognition Using Audio And Text (2018)18.02