Spoken Stereoset: On Evaluating Social Bias Toward Speaker In Speech Large Language Models

Abstract

Warning: This paper may contain texts with uncomfortable content. Large Language Models (LLMs) have achieved remarkable performance in various tasks, including those involving multimodal data like speech. However, these models often exhibit biases due to the nature of their training data. Recently, more Speech Large Language Models (SLLMs) have emerged, underscoring the urgent need to address these biases. This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in SLLMs. By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. Our experiments reveal significant insights into their performance and bias levels. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keylin2024spoken

Related papers

• Listen And Speak Fairly: A Study On Semantic Gender Bias In Speech Integrated Large Language Models (2024)6.34
• To Train Or Not To Train Adversarially: A Study Of Bias Mitigation Strategies For Speaker Recognition (2022)0.00
• Don't Speak Too Fast: The Impact Of Data Bias On Self-supervised Speech Models (2021)8.35
• Sonos Voice Control Bias Assessment Dataset: A Methodology For Demographic Bias Assessment In Voice Assistants (2024)0.00
• Demographic And Linguistic Bias Evaluation In Omnimodal Language Models (2026)0.00
• A Survey On Speech Large Language Models For Understanding (2024)4.52
• Speechllm-as-judges: Towards General And Interpretable Speech Quality Evaluation (2025)2.60
• Some Voices Are Too Common: Building Fair Speech Recognition Systems Using The Common Voice Dataset (2023)5.24