Integrating Pre-trained Speech And Language Models For End-to-end Speech Recognition

Abstract

Advances in machine learning have made it possible to perform various text and speech processing tasks, such as automatic speech recognition (ASR), in an end-to-end (E2E) manner. E2E approaches utilizing pre-trained models are gaining attention for conserving training data and resources. However, most of their applications in ASR involve only one of either a pre-trained speech or a language model. This paper proposes integrating a pre-trained speech representation model and a large language model (LLM) for E2E ASR. The proposed model enables the optimization of the entire ASR process, including acoustic feature extraction and acoustic and language modeling, by combining pre-trained models with a bridge network and also enables the application of remarkable developments in LLM utilization, such as parameter-efficient domain adaptation and inference optimization. Experimental results demonstrate that the proposed model achieves a performance comparable to that of modern E2E ASR models by

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhono2023integrating

Related papers

• Adapting Large Language Model With Speech For Fully Formatted End-to-end Speech Recognition (2023)0.00
• Knowledge Transfer From Large-scale Pretrained Language Models To End-to-end Speech Recognizers (2022)9.41
• Tiny-align: Bridging Automatic Speech Recognition And Large Language Model On The Edge (2024)0.00
• Delayed Fusion: Integrating Large Language Models Into First-pass Decoding In End-to-end Speech Recognition (2025)5.84
• A Comprehensive Solution To Connect Speech Encoder And Large Language Model For ASR (2024)0.00
• Speech-language Pre-training For End-to-end Spoken Language Understanding (2021)9.41
• Integrating Pretrained ASR And LM To Perform Sequence Generation For Spoken Language Understanding (2023)5.24
• Transfer Learning From Pre-trained Language Models Improves End-to-end Speech Summarization (2023)6.77