Intrinsicvoice: Empowering Llms With Intrinsic Real-time Voice Interaction Abilities

Abstract

Current methods of building LLMs with voice interaction capabilities rely heavily on explicit text autoregressive generation before or during speech response generation to maintain content quality, which unfortunately brings computational overhead and increases latency in multi-turn interactions. To address this, we introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. IntrinsicVoice aims to facilitate the transfer of textual capabilities of pre-trained LLMs to the speech modality by mitigating the modality gap between text and speech. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences while generating high-quality audio, significantly reducing the length difference between speech and text, speeding up inference, and alleviating long-text modeling issues. Additionally, we construct a multi-turn speech-to-speech dialogue dataset named \method-500k which includes nearly 500k turns of spee

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2024intrinsicvoice

Related papers

• Internalizing ASR With Implicit Chain Of Thought For Efficient Speech-to-speech Conversational LLM (2024)0.00
• Enhancing The Stability Of Llm-based Speech Generation Systems Through Self-supervised Representations (2024)0.00
• Recent Advances In Speech Language Models: A Survey (2024)14.64
• Boosting Large Language Model For Speech Synthesis: An Empirical Study (2023)6.77
• Voila: Voice-language Foundation Models For Real-time Autonomous Interaction And Voice Role-play (2025)0.00
• Funaudiollm: Voice Understanding And Generation Foundation Models For Natural Interaction Between Humans And Llms (2024)0.00
• Paralinguistics-enhanced Large Language Modeling Of Spoken Dialogue (2023)0.00
• Large Language Model Can Transcribe Speech In Multi-talker Scenarios With Versatile Instructions (2024)11.23