Investigating Training Strategies And Model Robustness Of Low-rank Adaptation For Language Modeling In Speech Recognition

Abstract

The use of low-rank adaptation (LoRA) with frozen pretrained language models (PLMs) has become increasing popular as a mainstream, resource-efficient modeling approach for memory-constrained hardware. In this study, we first explore how to enhance model performance by introducing various LoRA training strategies, achieving relative word error rate reductions of 3.50% on the public Librispeech dataset and of 3.67% on an internal dataset in the messaging domain. To further characterize the stability of LoRA-based second-pass speech recognition models, we examine robustness against input perturbations. These perturbations are rooted in homophone replacements and a novel metric called N-best Perturbation-based Rescoring Robustness (NPRR), both designed to measure the relative degradation in the performance of rescoring models. Our experimental results indicate that while advanced variants of LoRA, such as dynamic rank-allocated LoRA, lead to performance degradation in \(1\)-best perturbati

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyu2024investigating

Related papers

• Low-rank Adaptation Of Large Language Model Rescoring For Parameter-efficient Speech Recognition (2023)11.76
• Behind The Scenes: Mechanistic Interpretability Of Lora-adapted Whisper For Speech Emotion Recognition (2025)1.81
• Multimodal Large Language Models With Fusion Low Rank Adaptation For Device Directed Speech Detection (2024)0.00
• Dual-pipeline With Low-rank Adaptation For New Language Integration In Multilingual ASR (2024)3.58
• Full-rank No More: Low-rank Weight Training For Modern Speech Recognition Models (2024)2.26
• Speech Recognition With Llms Adapted To Disordered Speech Using Reinforcement Learning (2024)5.24
• Sparsely Shared Lora On Whisper For Child Speech Recognition (2023)9.59
• Gated Low-rank Adaptation For Personalized Code-switching Automatic Speech Recognition On The Low-spec Devices (2024)0.00