Effective Text Adaptation For Llm-based ASR Through Soft Prompt Fine-tuning

Abstract

The advent of Large Language Models (LLM) has reformed the Automatic Speech Recognition (ASR). Prompting LLM with audio embeddings to generate transcriptions becomes the new state-of-the-art ASR. Despite LLMs being trained with an extensive amount of text corpora, high-quality domain-specific text data can still significantly enhance ASR performance on domain adaptation tasks. Although LLM-based ASR can naturally incorporate more text corpora by fine-tuning the LLM decoder, fine-tuning such ASR on text-only data without paired prompts may diminish the effectiveness of domain-specific knowledge. To mitigate this issue, we propose a two-step soft prompt fine-tuning strategy that enhances domain-specific text adaptation. Experimental results show that text adaptation with our proposed method achieved a relative up to 9% Word Error Rate (WER) reduction and up to 18% Entity Error Rate (EER) reduction on the target domain compared to the baseline ASR. Combining this with domain-specific Lang

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyma2024effective

Related papers

• Prompting Large Language Models For Zero-shot Domain Adaptation In Speech Recognition (2023)0.00
• Chain-of-thought Prompting For Speech Translation (2024)6.34
• Towards Interfacing Large Language Models With ASR Systems Using Confidence Measures And Prompting (2024)7.16
• Corpus Synthesis For Zero-shot ASR Domain Adaptation Using Large Language Models (2023)5.84
• A Comprehensive Solution To Connect Speech Encoder And Large Language Model For ASR (2024)0.00
• Enhancing Large Language Model-based Speech Recognition By Contextualization For Rare And Ambiguous Words (2024)0.00
• Revise, Reason, And Recognize: Llm-based Emotion Recognition Via Emotion-specific Prompts And ASR Error Correction (2024)7.81
• Attention-based Contextual Language Model Adaptation For Speech Recognition (2021)0.00