Low-rank Adaptation Of Large Language Model Rescoring For Parameter-efficient Speech Recognition

Abstract

We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6.

Stats

• citations36
• S2 citations—
• github stars0
• HF likes0
• heat score11.76
• arxiv keyyu2023low

Related papers

• Investigating Training Strategies And Model Robustness Of Low-rank Adaptation For Language Modeling In Speech Recognition (2024)0.00
• Discriminative Speech Recognition Rescoring With Pre-trained Language Models (2023)2.26
• Dual-pipeline With Low-rank Adaptation For New Language Integration In Multilingual ASR (2024)3.58
• Lattice Rescoring Strategies For Long Short Term Memory Language Models In Speech Recognition (2017)9.76
• Audio-attention Discriminative Language Model For ASR Rescoring (2019)9.23
• Full-rank No More: Low-rank Weight Training For Modern Speech Recognition Models (2024)2.26
• Prompting Large Language Models For Zero-shot Domain Adaptation In Speech Recognition (2023)0.00
• Multimodal Large Language Models With Fusion Low Rank Adaptation For Device Directed Speech Detection (2024)0.00