Voiceagentbench: Are Voice Assistants Ready For Agentic Tasks?

Abstract

Large scale Speech Language Models have enabled voice assistants capable of understanding natural spoken queries and performing complex tasks. However, existing speech benchmarks largely focus on isolated capabilities such as transcription or question answering and do not systematically evaluate agentic behavior or adversarial robustness. To address this, we introduce VoiceAgentBench, a comprehensive benchmark for evaluating SpeechLMs in realistic spoken agentic settings, comprising 6,000+ synthetic spoken queries spanning single-tool invocations, multi-tool workflows, multi-turn dialogue, and safety evaluations across English and six Indic languages. To ensure speaker diversity, we further simulate speaker variability using a novel sampling strategy that selects audios for TTS voice conversion based on speaker embeddings to maximize acoustic diversity. Our evaluation measures tool selection accuracy, structural consistency, and the correctness of tool invocations, including adversaria

Stats

• citations0
• S2 citations—
• github stars3
• HF likes0
• heat score1.20
• arxiv keyjain2025voiceagentbench

Related papers

• Vocalbench: Benchmarking The Vocal Conversational Abilities For Speech Interaction Models (2025)0.00
• VCB Bench: An Evaluation Benchmark For Audio-grounded Large Language Model Conversational Agents (2025)0.00
• Audiotoolagent: An Agentic Framework For Audio-language Models (2025)2.60
• Spoken Conversational Agents With Large Language Models (2025)0.00
• Speaker Verification In Agent-generated Conversations (2024)0.00
• Audiobench: A Universal Benchmark For Audio Large Language Models (2024)10.21
• Speechrole: A Large-scale Dataset And Benchmark For Evaluating Speech Role-playing Agents (2025)1.91
• Mtavg-bench: A Comprehensive Benchmark For Evaluating Multi-talker Dialogue-centric Audio-video Generation (2026)0.00