Leveraging Weakly Supervised Data To Improve End-to-end Speech-to-text Translation

Abstract

End-to-end Speech Translation (ST) models have many potential advantages when compared to the cascade of Automatic Speech Recognition (ASR) and text Machine Translation (MT) models, including lowered inference latency and the avoidance of error compounding. However, the quality of end-to-end ST is often limited by a paucity of training data, since it is difficult to collect large parallel corpora of speech and translated transcript pairs. Previous studies have proposed the use of pre-trained components and multi-task learning in order to benefit from weakly supervised training data, such as speech-to-transcript or text-to-foreign-text pairs. In this paper, we demonstrate that using pre-trained MT or text-to-speech (TTS) synthesis models to convert weakly supervised data into speech-to-translation pairs for ST training can be more effective than multi-task learning. Furthermore, we demonstrate that a high quality end-to-end ST model can be trained using only weakly supervised datasets,

Stats

• citations54
• S2 citations—
• github stars0
• HF likes0
• heat score13.05
• arxiv keyjia2018leveraging

Related papers

• Leveraging Unsupervised And Weakly-supervised Data To Improve Direct Speech-to-speech Translation (2022)8.35
• Multilingual End-to-end Speech Translation (2019)0.00
• Data Efficient Direct Speech-to-text Translation With Modality Agnostic Meta-learning (2019)0.00
• Improving Cross-lingual Transfer Learning For End-to-end Speech Recognition With Speech Translation (2020)9.92
• Tackling Data Scarcity In Speech Translation Using Zero-shot Multilingual Machine Translation Techniques (2022)2.26
• Enhanced Direct Speech-to-speech Translation Using Self-supervised Pre-training And Data Augmentation (2022)10.85
• One-to-many Multilingual End-to-end Speech Translation (2019)9.23
• Generating Synthetic Speech From Spokenvocab For Speech Translation (2022)5.38