Beyond Silent Letters: Amplifying Llms In Emotion Recognition With Vocal Nuances

Abstract

Emotion recognition in speech is a challenging multimodal task that requires understanding both verbal content and vocal nuances. This paper introduces a novel approach to emotion detection using Large Language Models (LLMs), which have demonstrated exceptional capabilities in natural language understanding. To overcome the inherent limitation of LLMs in processing audio inputs, we propose SpeechCueLLM, a method that translates speech characteristics into natural language descriptions, allowing LLMs to perform multimodal emotion analysis via text prompts without any architectural changes. Our method is minimal yet impactful, outperforming baseline models that require structural modifications. We evaluate SpeechCueLLM on two datasets: IEMOCAP and MELD, showing significant improvements in emotion recognition accuracy, particularly for high-quality audio data. We also explore the effectiveness of various feature representations and fine-tuning strategies for different LLMs. Our experiment

Stats

• citations16
• S2 citations—
• github stars0
• HF likes0
• heat score9.23
• arxiv keywu2024beyond

Related papers

• Revise, Reason, And Recognize: Llm-based Emotion Recognition Via Emotion-specific Prompts And ASR Error Correction (2024)7.81
• Context And System Fusion In Post-asr Emotion Recognition With Large Language Models (2024)0.00
• LLM Supervised Pre-training For Multimodal Emotion Recognition In Conversations (2025)8.35
• Leveraging Label Potential For Enhanced Multimodal Emotion Recognition (2025)0.00
• Large Language Models Are Strong Audio-visual Speech Recognition Learners (2024)9.59
• Multistage Linguistic Conditioning Of Convolutional Layers For Speech Emotion Recognition (2021)9.23
• Whose Emotion Matters? Speaking Activity Localisation Without Prior Knowledge (2022)7.74
• Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-attention Cues In Multitask Learning (2024)0.00