LAMASSU: Streaming Language-agnostic Multilingual Speech Recognition And Translation Using Neural Transducers

Abstract

Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. It is thus possible to use a single transducer model to perform both tasks. In real-world applications, such joint ASR and ST models may need to be streaming and do not require source language identification (i.e. language-agnostic). In this paper, we propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers. Based on the transducer model structure, we propose four methods, a unified joint and prediction network for multilingual output, a clustered multilingual encoder, target language identification for encoder, and connectionist temporal classification regularization. Experimental results show that LAMASSU not only drastically reduces the model size but also reaches the performances of monolingual ASR and bilingual ST models.

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keywang2022lamassu

Related papers

• Large-scale Streaming End-to-end Speech Translation With Neural Transducers (2022)9.59
• Transducer-llama: Integrating Llms Into Streamable Transducer-based Speech Recognition (2024)3.58
• Label-synchronous Neural Transducer For E2E Simultaneous Speech Translation (2024)0.00
• Zero-resource Speech Translation And Recognition With Llms (2024)3.58
• Dual-decoder Transformer For Joint Automatic Speech Recognition And Multilingual Speech Translation (2020)13.73
• Improved Neural Language Model Fusion For Streaming Recurrent Neural Network Transducer (2020)8.82
• Leveraging Timestamp Information For Serialized Joint Streaming Recognition And Translation (2023)0.00
• Label-synchronous Neural Transducer For Adaptable Online E2E Speech Recognition (2023)3.58