Textless Direct Speech-to-speech Translation With Discrete Speech Representation

Abstract

Research on speech-to-speech translation (S2ST) has progressed rapidly in recent years. Many end-to-end systems have been proposed and show advantages over conventional cascade systems, which are often composed of recognition, translation and synthesis sub-systems. However, most of the end-to-end systems still rely on intermediate textual supervision during training, which makes it infeasible to work for languages without written forms. In this work, we propose a novel model, Textless Translatotron, which is based on Translatotron 2, for training an end-to-end direct S2ST model without any textual supervision. Instead of jointly training with an auxiliary task predicting target phonemes as in Translatotron 2, the proposed model uses an auxiliary task predicting discrete speech representations which are obtained from learned or random speech quantizers. When a speech encoder pre-trained with unsupervised speech data is used for both models, the proposed model obtains translation quality

Stats

• citations19
• S2 citations—
• github stars0
• HF likes0
• heat score9.76
• arxiv keyli2022textless

Related papers

• Direct Speech-to-speech Translation With Discrete Units (2021)13.97
• Leveraging Unsupervised And Weakly-supervised Data To Improve Direct Speech-to-speech Translation (2022)8.35
• Textless Speech-to-speech Translation On Real Data (2021)13.65
• Can We Achieve High-quality Direct Speech-to-speech Translation Without Parallel Speech Data? (2024)2.26
• Textless Speech-to-speech Translation With Limited Parallel Data (2023)3.58
• Translatotron 2: High-quality Direct Speech-to-speech Translation With Voice Preservation (2021)0.00
• Joint Pre-training With Speech And Bilingual Text For Direct Speech To Speech Translation (2022)7.81
• Transpeech: Speech-to-speech Translation With Bilateral Perturbation (2022)0.00