Closing The Gap Between Text And Speech Understanding In Llms

Abstract

Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts--and even cascaded pipelines--on language understanding tasks. We term this shortfall the text-speech understanding gap: the performance drop observed when a speech-adapted LLM processes spoken inputs relative to when the original text-based LLM processes the equivalent text. Recent approaches to narrowing this gap either rely on large-scale speech synthesis of text corpora, which is costly and heavily dependent on synthetic data, or on large-scale proprietary speech datasets, which are not reproducible. As a result, there remains a need for more data-efficient alternatives for closing the text-speech understanding gap. In this work, we analyze the gap as driven by two factors: (i) forgetting of text capabilities during adaptation, and (ii) cross-modal misalignment between speech and text. Based on thi

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keycuervo2025closing

Related papers

• A Survey On Speech Large Language Models For Understanding (2024)4.52
• Understanding Textual Capability Degradation In Speech Llms Via Parameter Importance Analysis (2025)0.00
• Recent Advances In Speech Language Models: A Survey (2024)14.64
• Exploring Fine-tuning Of Large Audio Language Models For Spoken Language Understanding Under Limited Speech Data (2025)0.00
• Hearing To Translate: The Effectiveness Of Speech Modality Integration Into Llms (2026)0.00
• Roadmap Towards Superhuman Speech Understanding Using Large Language Models (2024)0.00
• A Comprehensive Solution To Connect Speech Encoder And Large Language Model For ASR (2024)0.00
• Boosting Large Language Model For Speech Synthesis: An Empirical Study (2023)6.77