Translatotron 2: High-quality Direct Speech-to-speech Translation With Voice Preservation

Abstract

We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a linguistic decoder, an acoustic synthesizer, and a single attention module that connects them together. Experimental results on three datasets consistently show that Translatotron 2 outperforms the original Translatotron by a large margin on both translation quality (up to +15.5 BLEU) and speech generation quality, and approaches the same of cascade systems. In addition, we propose a simple method for preserving speakers' voices from the source speech to the translation speech in a different language. Unlike existing approaches, the proposed method is able to preserve each speaker's voice on speaker turns without requiring for speaker segmentation. Furthermore, compared to existing approaches, it better preserves speaker's privacy and mitigates potential misuse of voice cloning for creating spoofing audio artifacts.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyjia2021translatotron

Related papers

• Translatotron 3: Speech To Speech Translation With Monolingual Data (2023)8.09
• Textless Direct Speech-to-speech Translation With Discrete Speech Representation (2022)9.76
• Leveraging Unsupervised And Weakly-supervised Data To Improve Direct Speech-to-speech Translation (2022)8.35
• Transvip: Speech To Speech Translation System With Voice And Isochrony Preservation (2024)5.24
• Learning To Speak Fluently In A Foreign Language: Multilingual Speech Synthesis And Cross-language Voice Cloning (2019)15.03
• Deep Voice 2: Multi-speaker Neural Text-to-speech (2017)0.00
• Daspeech: Directed Acyclic Transformer For Fast And High-quality Speech-to-speech Translation (2023)5.24
• Transface: Unit-based Audio-visual Speech Synthesizer For Talking Head Translation (2023)7.16