Paralinguistics-aware Speech-empowered Large Language Models For Natural Conversation

Abstract

Recent work shows promising results in expanding the capabilities of large language models (LLM) to directly understand and synthesize speech. However, an LLM-based strategy for modeling spoken dialogs remains elusive, calling for further investigation. This paper introduces an extensive speech-text LLM framework, the Unified Spoken Dialog Model (USDM), designed to generate coherent spoken responses with naturally occurring prosodic features relevant to the given input speech without relying on explicit automatic speech recognition (ASR) or text-to-speech (TTS) systems. We have verified the inclusion of prosody in speech tokens that predominantly contain semantic information and have used this foundation to construct a prosody-infused speech-text model. Additionally, we propose a generalized speech-text pretraining scheme that enhances the capture of cross-modal semantics. To construct USDM, we fine-tune our speech-text model on spoken dialog data using a multi-step spoken dialog templ

Stats

• citations0
• S2 citations—
• github stars95
• HF likes0
• heat score3.96
• arxiv keykim2024paralinguistics

Related papers

• Paralinguistics-enhanced Large Language Modeling Of Spoken Dialogue (2023)0.00
• Discreteslu: A Large Language Model With Self-supervised Discrete Speech Units For Spoken Language Understanding (2024)5.84
• Prompting Large Language Models With Audio For General-purpose Speech Summarization (2024)6.34
• Recent Advances In Speech Language Models: A Survey (2024)14.64
• Large Language Model Can Transcribe Speech In Multi-talker Scenarios With Versatile Instructions (2024)11.23
• A Survey On Speech Large Language Models For Understanding (2024)4.52
• Discrete Multimodal Transformers With A Pretrained Large Language Model For Mixed-supervision Speech Processing (2024)0.00
• Augmenting Text For Spoken Language Understanding With Large Language Models (2023)0.00