E-chat: Emotion-sensitive Spoken Dialogue System With Large Language Models

Abstract

This study focuses on emotion-sensitive spoken dialogue in human-machine speech interaction. With the advancement of Large Language Models (LLMs), dialogue systems can handle multimodal data, including audio. Recent models have enhanced the understanding of complex audio signals through the integration of various audio events. However, they are unable to generate appropriate responses based on emotional speech. To address this, we introduce the Emotional chat Model (E-chat), a novel spoken dialogue system capable of comprehending and responding to emotions conveyed from speech. This model leverages an emotion embedding extracted by a speech encoder, combined with LLMs, enabling it to respond according to different emotional contexts. Additionally, we introduce the E-chat200 dataset, designed explicitly for emotion-sensitive spoken dialogue. In various evaluation metrics, E-chat consistently outperforms baseline model, demonstrating its potential in emotional comprehension and human-mac

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyxue2023e

Related papers

• Sd-eval: A Benchmark Dataset For Spoken Dialogue Understanding Beyond Words (2024)11.32
• Paralinguistics-enhanced Large Language Modeling Of Spoken Dialogue (2023)0.00
• Av-emodialog: Chat With Audio-visual Users Leveraging Emotional Cues (2024)0.00
• Beyond Silent Letters: Amplifying Llms In Emotion Recognition With Vocal Nuances (2024)9.23
• Paralinguistics-aware Speech-empowered Large Language Models For Natural Conversation (2024)3.96
• Audiochatllama: Towards General-purpose Speech Abilities For Llms (2023)9.41
• Context And System Fusion In Post-asr Emotion Recognition With Large Language Models (2024)0.00
• Revise, Reason, And Recognize: Llm-based Emotion Recognition Via Emotion-specific Prompts And ASR Error Correction (2024)7.81