Av-emodialog: Chat With Audio-visual Users Leveraging Emotional Cues
2024 · Se Jin Park, Yeonju Kim, Hyeongseop Rha, et al.
Abstract
In human communication, both verbal and non-verbal cues play a crucial role in conveying emotions, intentions, and meaning beyond words alone. These non-linguistic information, such as facial expressions, eye contact, voice tone, and pitch, are fundamental elements of effective interactions, enriching conversations by adding emotional and contextual depth. Recognizing the importance of non-linguistic content in communication, we present AV-EmoDialog, a dialogue system designed to exploit verbal and non-verbal information from users' audio-visual inputs to generate more responsive and empathetic interactions. AV-EmoDialog systematically exploits the emotional cues in audio-visual dialogues; extracting speech content and emotional tones from speech, analyzing fine-grained facial expressions from visuals, and integrating these cues to generate emotionally aware responses in an end-to-end manner. Through extensive experiments, we validate that the proposed AV-EmoDialog outperforms existing
Authors
(none)
Tags
Stats
Related papers
- Emogene: Audio-driven Emotional 3D Talking-head Generation (2024)2.26
- E-chat: Emotion-sensitive Spoken Dialogue System With Large Language Models (2023)7.50
- Reading The Mood Behind Words: Integrating Prosody-derived Emotional Context Into Socially Responsive VR Agents (2026)0.00
- Emotivetalk: Expressive Talking Head Generation Through Audio Information Decoupling And Emotional Video Diffusion (2024)0.00
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions Of Visual-audio Content (2024)10.48
- AV2AV: Direct Audio-visual Speech To Audio-visual Speech Translation With Unified Audio-visual Speech Representation (2023)6.77
- Advancing User-voice Interaction: Exploring Emotion-aware Voice Assistants Through A Role-swapping Approach (2025)6.77
- Qieemo: Speech Is All You Need In The Emotion Recognition In Conversations (2025)0.00