Audiobench: A Universal Benchmark For Audio Large Language Models

Abstract

We introduce AudioBench, a universal benchmark designed to evaluate Audio Large Language Models (AudioLLMs). It encompasses 8 distinct tasks and 26 datasets, among which, 7 are newly proposed datasets. The evaluation targets three main aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic). Despite recent advancements, there lacks a comprehensive benchmark for AudioLLMs on instruction following capabilities conditioned on audio signals. AudioBench addresses this gap by setting up datasets as well as desired evaluation metrics. Besides, we also evaluated the capabilities of five popular models and found that no single model excels consistently across all tasks. We outline the research outlook for AudioLLMs and anticipate that our open-sourced evaluation toolkit, data, and leaderboard will offer a robust testbed for future model developments.

Stats

• citations22
• S2 citations—
• github stars0
• HF likes0
• heat score10.21
• arxiv keywang2024audiobench

Related papers

• VCB Bench: An Evaluation Benchmark For Audio-grounded Large Language Model Conversational Agents (2025)0.00
• Towards Holistic Evaluation Of Large Audio-language Models: A Comprehensive Survey (2026)9.75
• Vocalbench: Benchmarking The Vocal Conversational Abilities For Speech Interaction Models (2025)0.00
• Larabench: Benchmarking Arabic AI With Large Language Models (2023)6.77
• Measuring Audio's Impact On Correctness: Audio-contribution-aware Post-training Of Large Audio Language Models (2025)0.00
• Roadmap Towards Superhuman Speech Understanding Using Large Language Models (2024)0.00
• All That Glitters Is Not Audio: Rethinking Text Priors And Audio Reliance In Audio-language Evaluation (2026)0.00
• MMSU: A Massive Multi-task Spoken Language Understanding And Reasoning Benchmark (2025)2.29