Analyzing ASR Pretraining For Low-resource Speech-to-text Translation

Abstract

Previous work has shown that for low-resource source languages, automatic speech-to-text translation (AST) can be improved by pretraining an end-to-end model on automatic speech recognition (ASR) data from a high-resource language. However, it is not clear what factors --e.g., language relatedness or size of the pretraining data-- yield the biggest improvements, or whether pretraining can be effectively combined with other methods such as data augmentation. Here, we experiment with pretraining on datasets of varying sizes, including languages related and unrelated to the AST source language. We find that the best predictor of final AST performance is the word error rate of the pretrained ASR model, and that differences in ASR/AST performance correlate with how phonetic information is encoded in the later RNN layers of our model. We also show that pretraining and data augmentation yield complementary benefits for AST.

Stats

• citations21
• S2 citations—
• github stars0
• HF likes0
• heat score10.07
• arxiv keystoian2019analyzing

Related papers

• Pretraining By Backtranslation For End-to-end ASR In Low-resource Settings (2018)0.00
• Harnessing Indirect Training Data For End-to-end Automatic Speech Translation: Tricks Of The Trade (2019)0.00
• Strategies For Improving Low Resource Speech To Text Translation Relying On Pre-trained ASR Models (2023)5.24
• Pre-training End-to-end ASR Models With Augmented Speech Samples Queried By Text (2023)0.00
• Improving Cross-lingual Transfer Learning For End-to-end Speech Recognition With Speech Translation (2020)9.92
• ASR Data Augmentation In Low-resource Settings Using Cross-lingual Multi-speaker TTS And Cross-lingual Voice Conversion (2022)6.77
• Reduce, Reuse, Recycle: Is Perturbed Data Better Than Other Language Augmentation For Low Resource Self-supervised Speech Models (2023)0.00
• Adaptive Activation Network For Low Resource Multilingual Speech Recognition (2022)0.00