Multilingual And Fully Non-autoregressive ASR With Large Language Model Fusion: A Comprehensive Study

Abstract

In the era of large models, the autoregressive nature of decoding often results in latency serving as a significant bottleneck. We propose a non-autoregressive LM-fused ASR system that effectively leverages the parallelization capabilities of accelerator hardware. Our approach combines the Universal Speech Model (USM) and the PaLM 2 language model in per-segment scoring mode, achieving an average relative WER improvement across all languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our comprehensive ablation study analyzes key parameters such as LLM size, context length, vocabulary size, fusion methodology. For instance, we explore the impact of LLM size ranging from 128M to 340B parameters on ASR performance. This study provides valuable insights into the factors influencing the effectiveness of practical large-scale LM-fused speech recognition systems.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhuang2024multilingual

Related papers

• Delayed Fusion: Integrating Large Language Models Into First-pass Decoding In End-to-end Speech Recognition (2025)5.84
• It's Never Too Late: Fusing Acoustic Information Into Large Language Models For Automatic Speech Recognition (2024)0.00
• Context And System Fusion In Post-asr Emotion Recognition With Large Language Models (2024)0.00
• Transfer Learning Of Language-independent End-to-end ASR With Language Model Fusion (2018)0.00
• Let's Fuse Step By Step: A Generative Fusion Decoding Algorithm With Llms For Robust And Instruction-aware ASR And OCR (2024)0.00
• Improving Tail Performance Of A Deliberation E2E ASR Model Using A Large Text Corpus (2020)10.21
• A Comprehensive Solution To Connect Speech Encoder And Large Language Model For ASR (2024)0.00
• Multimodal Large Language Models With Fusion Low Rank Adaptation For Device Directed Speech Detection (2024)0.00