Multi-modal Transformers Utterance-level Code-switching Detection
2020 · Krishna D N
Abstract
An utterance that contains speech from multiple languages is known as a code-switched sentence. In this work, we propose a novel technique to predict whether given audio is mono-lingual or code-switched. We propose a multi-modal learning approach by utilising the phoneme information along with audio features for code-switch detection. Our model consists of a Phoneme Network that processes phoneme sequence and Audio Network(AN), which processes the mfcc features. We fuse representation learned from both the Networks to predict if the utterance is code-switched or not. The Audio Network and Phonetic Network consist of initial convolution, Bi-LSTM, and transformer encoder layers. The transformer encoder layer helps in selecting important and relevant features for better classification by using self-attention. We show that utilising the phoneme sequence of the utterance along with the mfcc features improves the performance of code-switch detection significantly. We train and evaluate our m
Authors
(none)
Tags
Stats
Related papers
- Transformer-transducers For Code-switched Speech Recognition (2020)10.97
- Unified Model For Code-switching Speech Recognition And Language Identification Based On A Concatenated Tokenizer (2023)8.09
- Code-switched Language Models Using Neural Based Synthetic Data From Parallel Sentences (2019)11.29
- Joint Modeling Of Code-switched And Monolingual ASR Via Conditional Factorization (2021)8.60
- Audioformer: Audio Transformer Learns Audio Feature Representations From Discrete Acoustic Codes (2023)0.00
- Decoupling Pronunciation And Language For End-to-end Code-switching Automatic Speech Recognition (2020)0.00
- Using Heterogeneity In Semi-supervised Transcription Hypotheses To Improve Code-switched Speech Recognition (2021)0.00
- Multilingual Self-supervised Speech Representations Improve The Speech Recognition Of Low-resource African Languages With Codeswitching (2023)0.00