Dual-pipeline With Low-rank Adaptation For New Language Integration In Multilingual ASR

Abstract

This paper addresses challenges in integrating new languages into a pre-trained multilingual automatic speech recognition (mASR) system, particularly in scenarios where training data for existing languages is limited or unavailable. The proposed method employs a dual-pipeline with low-rank adaptation (LoRA). It maintains two data flow pipelines-one for existing languages and another for new languages. The primary pipeline follows the standard flow through the pre-trained parameters of mASR, while the secondary pipeline additionally utilizes language-specific parameters represented by LoRA and a separate output decoder module. Importantly, the proposed approach minimizes the performance degradation of existing languages and enables a language-agnostic operation mode, facilitated by a decoder selection strategy. We validate the effectiveness of the proposed method by extending the pre-trained Whisper model to 19 new languages from the FLEURS dataset

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keykhassanov2024dual

Related papers

• Investigating Training Strategies And Model Robustness Of Low-rank Adaptation For Language Modeling In Speech Recognition (2024)0.00
• Transfer Learning Of Language-independent End-to-end ASR With Language Model Fusion (2018)0.00
• Multimodal Large Language Models With Fusion Low Rank Adaptation For Device Directed Speech Detection (2024)0.00
• Low-rank Adaptation Of Large Language Model Rescoring For Parameter-efficient Speech Recognition (2023)11.76
• Hdmole: Mixture Of Lora Experts With Hierarchical Routing And Dynamic Thresholds For Fine-tuning Llm-based ASR Models (2024)8.09
• A Domain Adaptation Framework For Speech Recognition Systems With Only Synthetic Data (2025)5.24
• Whisper-lm: Improving ASR Models With Language Models For Low-resource Languages (2025)3.29
• Behind The Scenes: Mechanistic Interpretability Of Lora-adapted Whisper For Speech Emotion Recognition (2025)1.81