Delayed Fusion: Integrating Large Language Models Into First-pass Decoding In End-to-end Speech Recognition

Abstract

This paper presents an efficient decoding approach for end-to-end automatic speech recognition (E2E-ASR) with large language models (LLMs). Although shallow fusion is the most common approach to incorporate language models into E2E-ASR decoding, we face two practical problems with LLMs. (1) LLM inference is computationally costly. (2) There may be a vocabulary mismatch between the ASR model and the LLM. To resolve this mismatch, we need to retrain the ASR model and/or the LLM, which is at best time-consuming and in many cases not feasible. We propose "delayed fusion," which applies LLM scores to ASR hypotheses with a delay during decoding and enables easier use of pre-trained LLMs in ASR tasks. This method can reduce not only the number of hypotheses scored by the LLM but also the number of LLM inference calls. It also allows re-tokenizion of ASR hypotheses during decoding if ASR and LLM employ different tokenizations. We demonstrate that delayed fusion provides improved decoding speed

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyhori2025delayed

Related papers

• Multilingual And Fully Non-autoregressive ASR With Large Language Model Fusion: A Comprehensive Study (2024)0.00
• It's Never Too Late: Fusing Acoustic Information Into Large Language Models For Automatic Speech Recognition (2024)0.00
• Integrating Pre-trained Speech And Language Models For End-to-end Speech Recognition (2023)0.00
• Let's Fuse Step By Step: A Generative Fusion Decoding Algorithm With Llms For Robust And Instruction-aware ASR And OCR (2024)0.00
• Adapting Large Language Model With Speech For Fully Formatted End-to-end Speech Recognition (2023)0.00
• Investigating Decoder-only Large Language Models For Speech-to-text Translation (2024)0.00
• On Decoder-only Architecture For Speech-to-text And Large Language Model Integration (2023)0.00
• A Comprehensive Solution To Connect Speech Encoder And Large Language Model For ASR (2024)0.00