Audiochatllama: Towards General-purpose Speech Abilities For Llms

Abstract

In this work, we extend the instruction-tuned Llama-2 model with end-to-end general-purpose speech processing and reasoning abilities while maintaining the wide range of original LLM capabilities, without using any carefully curated paired data. The resulting end-to-end model, named AudioChatLlama, can utilize audio prompts as a replacement for text and sustain a conversation. Such a model also has extended cross-modal capabilities such as being able to perform spoken question answering (QA), speech translation, and audio summarization amongst many other closed and open-domain tasks. This is unlike prior approaches in speech, in which LLMs are extended to handle audio for a limited number of pre-designated tasks. On both synthesized and recorded speech QA test sets, evaluations show that our end-to-end approach is on par with or outperforms cascaded systems (speech recognizer + LLM) in terms of modeling the response to a prompt. Furthermore, unlike cascades, our approach can interchang

Stats

• citations17
• S2 citations—
• github stars0
• HF likes0
• heat score9.41
• arxiv keyfathullah2023audiochatllama

Related papers

• Large Language Models Are Strong Audio-visual Speech Recognition Learners (2024)9.59
• Prompting Large Language Models With Audio For General-purpose Speech Summarization (2024)6.34
• Get Large Language Models Ready To Speak: A Late-fusion Approach For Speech Generation (2024)5.24
• Llama-omni2: Llm-based Real-time Spoken Chatbot With Autoregressive Streaming Speech Synthesis (2025)6.77
• Spoken Conversational Agents With Large Language Models (2025)0.00
• Videollama 2: Advancing Spatial-temporal Modeling And Audio Understanding In Video-llms (2024)0.00
• Large Language Model Can Transcribe Speech In Multi-talker Scenarios With Versatile Instructions (2024)11.23
• SELMA: A Speech-enabled Language Model For Virtual Assistant Interactions (2025)2.26