The Speech-llm Takes It All: A Truly Fully End-to-end Spoken Dialogue State Tracking Approach

Abstract

This paper presents a comparative study of context management strategies for end-to-end Spoken Dialog State Tracking using Speech-LLMs. We systematically evaluate traditional multimodal context (combining text history and spoken current turn), full spoken history, and compressed spoken history approaches. Our experiments on the SpokenWOZ corpus demonstrate that providing the full spoken conversation as input yields the highest performance among models of similar size, significantly surpassing prior methods. Furthermore, we show that attention-pooling-based compression of the spoken history offers a strong trade-off, maintaining competitive accuracy with reduced context size. Detailed analysis confirms that improvements stem from more effective context utilization.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyghazal2025the

Related papers

• Attentive Contextual Carryover For Multi-turn End-to-end Spoken Language Understanding (2021)7.16
• End-to-end Speech Recognition Contextualization With Large Language Models (2023)0.00
• Where Visual Speech Meets Language: VSP-LLM Framework For Efficient And Context-aware Visual Speech Processing (2024)0.00
• Spokenwoz: A Large-scale Speech-text Benchmark For Spoken Task-oriented Dialogue Agents (2023)2.26
• Improving Audio Codec-based Zero-shot Text-to-speech Synthesis With Multi-modal Context And Large Language Model (2024)2.26
• Hearing To Translate: The Effectiveness Of Speech Modality Integration Into Llms (2026)0.00
• Context And System Fusion In Post-asr Emotion Recognition With Large Language Models (2024)0.00
• Dynamically Context-sensitive Time-decay Attention For Dialogue Modeling (2018)2.26