Developing A Multilingual Dataset And Evaluation Metrics For Code-switching: A Focus On Hong Kong's Polylingual Dynamics

Abstract

The existing audio datasets are predominantly tailored towards single languages, overlooking the complex linguistic behaviors of multilingual communities that engage in code-switching. This practice, where individuals frequently mix two or more languages in their daily interactions, is particularly prevalent in multilingual regions such as Hong Kong, China. To bridge this gap, we have developed a 34.8-hour dataset of Mixed Cantonese and English (MCE) audio using our Multi-Agent Data Generation Framework (MADGF). We fine-tuned the open-source multilingual Automatic Speech Recognition (ASR) model, Whisper, with the MCE dataset, leading to impressive zero-shot performance. The traditional metrics overlook important factors such as latency in real-world applications and code-switching scenarios. We have introduced a novel evaluation metric called Fidelity to the Original Audio, Accuracy, and Latency (FAL). This metric aims to overcome the limitations of traditional metrics used to assess A

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyxie2023developing

Related papers

• Benchmarking Evaluation Metrics For Code-switching Automatic Speech Recognition (2022)5.84
• Switchlingua: The First Large-scale Multilingual And Multi-ethnic Code-switching Dataset (2025)0.00
• Integrating Knowledge In End-to-end Automatic Speech Recognition For Mandarin-english Code-switching (2021)5.24
• Language Modeling For Code-switching: Evaluation, Integration Of Monolingual Data, And Discriminative Training (2018)5.24
• Merlion CCS Challenge: A English-mandarin Code-switching Child-directed Speech Corpus For Language Identification And Diarization (2023)0.00
• The ASRU 2019 Mandarin-english Code-switching Speech Recognition Challenge: Open Datasets, Tracks, Methods And Results (2020)0.00
• Using Heterogeneity In Semi-supervised Transcription Hypotheses To Improve Code-switched Speech Recognition (2021)0.00
• Hike: Hierarchical Evaluation Framework For Korean-english Code-switching Speech Recognition (2025)2.08