Costa: Code-switched Speech Translation Using Aligned Speech-text Interleaving

Abstract

Code-switching is a widely prevalent linguistic phenomenon in multilingual societies like India. Building speech-to-text models for code-switched speech is challenging due to limited availability of datasets. In this work, we focus on the problem of spoken translation (ST) of code-switched speech in Indian languages to English text. We present a new end-to-end model architecture COSTA that scaffolds on pretrained automatic speech recognition (ASR) and machine translation (MT) modules (that are more widely available for many languages). Speech and ASR text representations are fused using an aligned interleaving scheme and are fed further as input to a pretrained MT module; the whole pipeline is then trained end-to-end for spoken translation using synthetically created ST data. We also release a new evaluation benchmark for code-switched Bengali-English, Hindi-English, Marathi-English and Telugu- English speech to English text. COSTA significantly outperforms many competitive cascaded an

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyshankar2024costa

Related papers

• Unified Model For Code-switching Speech Recognition And Language Identification Based On A Concatenated Tokenizer (2023)8.09
• Code-switching Without Switching: Language Agnostic End-to-end Speech Translation (2022)0.00
• Balanced End-to-end Monolingual Pre-training For Low-resourced Indic Languages Code-switching Speech Recognition (2021)0.00
• End-to-end ASR For Code-switched Hindi-english Speech (2019)0.00
• Code-switching Speech Recognition Under The Lens: Model- And Data-centric Perspectives (2025)0.00
• Towards End-to-end Code-switching Speech Recognition (2018)0.00
• Transformer-transducers For Code-switched Speech Recognition (2020)10.97
• Synchronous Speech Recognition And Speech-to-text Translation With Interactive Decoding (2019)10.48