Speechllm-as-judges: Towards General And Interpretable Speech Quality Evaluation
2025 · Hui Wang, Jinghua Zhao, Yifan Yang, et al.
Abstract
Generative speech technologies are progressing rapidly, but evaluating the perceptual quality of synthetic speech remains a core challenge. Existing methods typically rely on scalar scores or binary decisions, which lack interpretability and generalization across tasks and languages. We present SpeechLLM-as-Judges, a new paradigm for enabling large language models (LLMs) to conduct structured and explanation-based speech quality evaluation. To support this direction, we introduce SpeechEval, a large-scale dataset containing 32,207 multilingual speech clips and 128,754 annotations spanning four tasks: quality assessment, pairwise comparison, improvement suggestion, and deepfake detection. Based on this resource, we develop SQ-LLM, a speech-quality-aware LLM trained with chain-of-thought reasoning and reward optimization to improve capability. Experimental results show that SQ-LLM delivers strong performance across tasks and languages, revealing the potential of this paradigm for advanci
Authors
(none)
Tags
Stats
Related papers
- A Survey On Speech Large Language Models For Understanding (2024)4.52
- Vocalbench: Benchmarking The Vocal Conversational Abilities For Speech Interaction Models (2025)0.00
- Audiochatllama: Towards General-purpose Speech Abilities For Llms (2023)9.41
- Towards Holistic Evaluation Of Large Audio-language Models: A Comprehensive Survey (2026)9.75
- Squid: Measuring Speech Naturalness In Many Languages (2022)9.41
- Hearing To Translate: The Effectiveness Of Speech Modality Integration Into Llms (2026)0.00
- Large Language Model Based Generative Error Correction: A Challenge And Baselines For Speech Recognition, Speaker Tagging, And Emotion Recognition (2024)7.81
- Recent Advances In Speech Language Models: A Survey (2024)14.64