MMSU: A Massive Multi-task Spoken Language Understanding And Reasoning Benchmark

Abstract

Speech inherently contains rich acoustic information that extends far beyond the textual language. In real-world spoken language understanding, effective interpretation often requires integrating semantic meaning (e.g., content), paralinguistic features (e.g., emotions, speed, pitch) and phonological characteristics (e.g., prosody, intonation, rhythm), which are embedded in speech. While recent multimodal Speech Large Language Models (SpeechLLMs) have demonstrated remarkable capabilities in processing audio information, their ability to perform fine-grained perception and complex reasoning in natural speech remains largely unexplored. To address this gap, we introduce MMSU, a comprehensive benchmark designed specifically for understanding and reasoning in spoken language. MMSU comprises 5,000 meticulously curated audio-question-answer triplets across 47 distinct tasks. To ground our benchmark in linguistic theory, we systematically incorporate a wide range of linguistic phenomena, incl

Stats

• citations0
• S2 citations—
• github stars13
• HF likes0
• heat score2.29
• arxiv keywang2025mmsu

Related papers

• Audiobench: A Universal Benchmark For Audio Large Language Models (2024)10.21
• SLUE Phase-2: A Benchmark Suite Of Diverse Spoken Language Understanding Tasks (2022)10.07
• A Survey On Speech Large Language Models For Understanding (2024)4.52
• Sd-eval: A Benchmark Dataset For Spoken Dialogue Understanding Beyond Words (2024)11.32
• Omhbench: Benchmarking Balanced And Grounded Omni-modal Multi-hop Reasoning (2026)0.00
• Roadmap Towards Superhuman Speech Understanding Using Large Language Models (2024)0.00
• Vocalbench: Benchmarking The Vocal Conversational Abilities For Speech Interaction Models (2025)0.00
• Dynamic-superb Phase-2: A Collaboratively Expanding Benchmark For Measuring The Capabilities Of Spoken Language Models With 180 Tasks (2024)4.61