Generating Data With Text-to-speech And Large-language Models For Conversational Speech Recognition

Abstract

Currently, a common approach in many speech processing tasks is to leverage large scale pre-trained models by fine-tuning them on in-domain data for a particular application. Yet obtaining even a small amount of such data can be problematic, especially for sensitive domains and conversational speech scenarios, due to both privacy issues and annotation costs. To address this, synthetic data generation using single speaker datasets has been employed. Yet, for multi-speaker cases, such an approach often requires extensive manual effort and is prone to domain mismatches. In this work, we propose a synthetic data generation pipeline for multi-speaker conversational ASR, leveraging a large language model (LLM) for content creation and a conversational multi-speaker text-to-speech (TTS) model for speech synthesis. We conduct evaluation by fine-tuning the Whisper ASR model for telephone and distant conversational speech settings, using both in-domain data and generated synthetic data. Our resu

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keycornell2024generating

Related papers

• A Framework For Synthetic Audio Conversations Generation Using Large Language Models (2024)3.58
• Generating Synthetic Audio Data For Attention-based Speech Recognition Systems (2019)12.68
• Instruction Data Generation And Unsupervised Adaptation For Speech Language Models (2024)3.58
• Corpus Synthesis For Zero-shot ASR Domain Adaptation Using Large Language Models (2023)5.84
• A Domain Adaptation Framework For Speech Recognition Systems With Only Synthetic Data (2025)5.24
• Training Data Augmentation For Dysarthric Automatic Speech Recognition By Text-to-dysarthric-speech Synthesis (2024)10.48
• On The Effect Of Purely Synthetic Training Data For Different Automatic Speech Recognition Architectures (2024)0.00
• On The Problem Of Text-to-speech Model Selection For Synthetic Data Generation In Automatic Speech Recognition (2024)4.52