Instruction Data Generation And Unsupervised Adaptation For Speech Language Models

Abstract

In this paper, we propose three methods for generating synthetic samples to train and evaluate multimodal large language models capable of processing both text and speech inputs. Addressing the scarcity of samples containing both modalities, synthetic data generation emerges as a crucial strategy to enhance the performance of such systems and facilitate the modeling of cross-modal relationships between the speech and text domains. Our process employs large language models to generate textual components and text-to-speech systems to generate speech components. The proposed methods offer a practical and effective means to expand the training dataset for these models. Experimental results show progress in achieving an integrated understanding of text and speech. We also highlight the potential of using unlabeled speech data to generate synthetic samples comparable in quality to those with available transcriptions, enabling the expansion of these models to more languages.

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keynoroozi2024instruction

Related papers

• Multimodal Speech Synthesis Architecture For Unsupervised Speaker Adaptation (2018)6.34
• Generating Data With Text-to-speech And Large-language Models For Conversational Speech Recognition (2024)6.34
• Data Augmentation For Spoken Language Understanding Via Joint Variational Generation (2018)10.61
• Fake It To Make It: Using Synthetic Data To Remedy The Data Shortage In Joint Multimodal Speech-and-gesture Synthesis (2024)6.34
• From Alignment To Advancement: Bootstrapping Audio-language Alignment With Synthetic Data (2025)2.26
• Using Speech Synthesis To Train End-to-end Spoken Language Understanding Models (2019)9.23
• Data Augmentation For Spoken Language Understanding Via Pretrained Language Models (2020)0.00
• Unsupervised Data Validation Methods For Efficient Model Training (2024)0.00