Leveraging Unsupervised And Weakly-supervised Data To Improve Direct Speech-to-speech Translation

Abstract

End-to-end speech-to-speech translation (S2ST) without relying on intermediate text representations is a rapidly emerging frontier of research. Recent works have demonstrated that the performance of such direct S2ST systems is approaching that of conventional cascade S2ST when trained on comparable datasets. However, in practice, the performance of direct S2ST is bounded by the availability of paired S2ST training data. In this work, we explore multiple approaches for leveraging much more widely available unsupervised and weakly-supervised speech and text data to improve the performance of direct S2ST based on Translatotron 2. With our most effective approaches, the average translation quality of direct S2ST on 21 language pairs on the CVSS-C corpus is improved by +13.6 BLEU (or +113% relatively), as compared to the previous state-of-the-art trained without additional data. The improvements on low-resource language are even more significant (+398% relatively on average). Our comparativ

Stats

• citations12
• S2 citations—
• github stars0
• HF likes0
• heat score8.35
• arxiv keyjia2022leveraging

Related papers

• Leveraging Weakly Supervised Data To Improve End-to-end Speech-to-text Translation (2018)13.05
• Enhanced Direct Speech-to-speech Translation Using Self-supervised Pre-training And Data Augmentation (2022)10.85
• Joint Pre-training With Speech And Bilingual Text For Direct Speech To Speech Translation (2022)7.81
• Textless Direct Speech-to-speech Translation With Discrete Speech Representation (2022)9.76
• Leveraging Pseudo-labeled Data To Improve Direct Speech-to-speech Translation (2022)10.33
• Textless Speech-to-speech Translation On Real Data (2021)13.65
• Translatotron 3: Speech To Speech Translation With Monolingual Data (2023)8.09
• Improving Cascaded Unsupervised Speech Translation With Denoising Back-translation (2023)0.00