Enhancing Multilingual Speech Recognition Through Language Prompt Tuning And Frame-level Language Adapter

Abstract

Multilingual intelligent assistants, such as ChatGPT, have recently gained popularity. To further expand the applications of multilingual artificial intelligence assistants and facilitate international communication, it is essential to enhance the performance of multilingual speech recognition, which is a crucial component of speech interaction. In this paper, we propose two simple and parameter-efficient methods: language prompt tuning and frame-level language adapter, to respectively enhance language-configurable and language-agnostic multilingual speech recognition. Additionally, we explore the feasibility of integrating these two approaches using parameter-efficient fine-tuning methods. Our experiments demonstrate significant performance improvements across seven languages using our proposed methods.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2023enhancing

Related papers

• Unipet-spk: A Unified Framework For Parameter-efficient Tuning Of Pre-trained Speech Models For Robust Speaker Verification (2025)4.52
• Rapid Language Adaptation For Multilingual E2E Speech Recognition Using Encoder Prompting (2024)0.00
• Intapt: Information-theoretic Adversarial Prompt Tuning For Enhanced Non-native Speech Recognition (2023)3.58
• Leveraging Parameter-efficient Transfer Learning For Multi-lingual Text-to-speech Adaptation (2024)0.00
• Elp-adapters: Parameter Efficient Adapter Tuning For Various Speech Processing Tasks (2024)7.81
• Speechgen: Unlocking The Generative Power Of Speech Language Models With Prompts (2023)0.00
• Adapt-and-adjust: Overcoming The Long-tail Problem Of Multilingual Speech Recognition (2020)10.35
• Effective Text Adaptation For Llm-based ASR Through Soft Prompt Fine-tuning (2024)5.84