Language-universal Adapter Learning With Knowledge Distillation For End-to-end Multilingual Speech Recognition

Abstract

In this paper, we propose a language-universal adapter learning framework based on a pre-trained model for end-to-end multilingual automatic speech recognition (ASR). For acoustic modeling, the wav2vec 2.0 pre-trained model is fine-tuned by inserting language-specific and language-universal adapters. An online knowledge distillation is then used to enable the language-universal adapters to learn both language-specific and universal features. The linguistic information confusion is also reduced by leveraging language identifiers (LIDs). With LIDs we perform a position-wise modification on the multi-head attention outputs. In the inference procedure, the language-specific adapters are removed while the language-universal adapters are kept activated. The proposed method improves the recognition accuracy and addresses the linear increase of the number of adapters' parameters with the number of languages in common multilingual ASR systems. Experiments on the BABEL dataset confirm the effect

Stats

• citations0
• S2 citations—
• github stars2
• HF likes0
• heat score0.95
• arxiv keyshen2023language

Related papers

• Massively Multilingual Adversarial Speech Recognition (2019)11.93
• Multilingual Speech Recognition Using Knowledge Transfer Across Learning Processes (2021)0.00
• Transfer Learning Of Language-independent End-to-end ASR With Language Model Fusion (2018)0.00
• Efficient Adapter Transfer Of Self-supervised Speech Models For Automatic Speech Recognition (2022)12.68
• BLSP-KD: Bootstrapping Language-speech Pre-training Via Knowledge Distillation (2024)0.00
• Knowledge Distillation From Language Model To Acoustic Model: A Hierarchical Multi-task Learning Approach (2021)3.58
• An Adapter-based Unified Model For Multiple Spoken Language Processing Tasks (2024)0.00
• An Adapter Based Pre-training For Efficient And Scalable Self-supervised Speech Representation Learning (2021)8.35