Speechgpt: Empowering Large Language Models With Intrinsic Cross-modal Conversational Abilities

Abstract

Multi-modal large language models are regarded as a crucial step towards Artificial General Intelligence (AGI) and have garnered significant interest with the emergence of ChatGPT. However, current speech-language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. In this paper, we propose SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, capable of perceiving and generating multi-model content. With discrete speech representations, we first construct SpeechInstruct, a large-scale cross-modal speech instruction dataset. Additionally, we employ a three-stage training strategy that includes modality-adaptation pre-training, cross-modal instruction fine-tuning, and chain-of-modality instruction fine-tuning. The experimental results demonstrate that SpeechGPT has an impressive capacity to follow multi-modal human instructions and highlight the potential of handling multiple modalities with one model. Demos are shown

Stats

• citations162
• S2 citations—
• github stars0
• HF likes0
• heat score16.59
• arxiv keyzhang2023speechgpt

Related papers

• Chatbridge: Bridging Modalities With Large Language Model As A Language Catalyst (2023)0.00
• X-LLM: Bootstrapping Advanced Large Language Models By Treating Multi-modalities As Foreign Languages (2023)0.00
• Anygpt: Unified Multimodal LLM With Discrete Sequence Modeling (2024)0.00
• Paralinguistics-enhanced Large Language Modeling Of Spoken Dialogue (2023)0.00
• A Review Of Multi-modal Large Language And Vision Models (2024)0.00
• Towards Multi-modal Mastery: A 4.5B Parameter Truly Multi-modal Small Language Model (2024)2.26
• Teaching A Multilingual Large Language Model To Understand Multilingual Speech Via Multi-instructional Training (2024)0.00
• Speechgen: Unlocking The Generative Power Of Speech Language Models With Prompts (2023)0.00