Styles2st: Zero-shot Style Transfer For Direct Speech-to-speech Translation

Abstract

Direct speech-to-speech translation (S2ST) has gradually become popular as it has many advantages compared with cascade S2ST. However, current research mainly focuses on the accuracy of semantic translation and ignores the speech style transfer from a source language to a target language. The lack of high-fidelity expressive parallel data makes such style transfer challenging, especially in more practical zero-shot scenarios. To solve this problem, we first build a parallel corpus using a multi-lingual multi-speaker text-to-speech synthesis (TTS) system and then propose the StyleS2ST model with cross-lingual speech style transfer ability based on a style adaptor on a direct S2ST system framework. Enabling continuous style space modeling of an acoustic model through parallel corpus training and non-parallel TTS data augmentation, StyleS2ST captures cross-lingual acoustic feature mapping from the source to the target language. Experiments show that StyleS2ST achieves good style similarit

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysong2023styles2st

Related papers

• Speech-to-speech Translation With Discrete-unit-based Style Transfer (2023)0.00
• MSLM-S2ST: A Multitask Speech Language Model For Textless Speech-to-speech Translation With Speaker Style Preservation (2024)0.00
• Textless And Non-parallel Speech-to-speech Emotion Style Transfer (2025)0.00
• Joint Pre-training With Speech And Bilingual Text For Direct Speech To Speech Translation (2022)7.81
• Styletts 2: Towards Human-level Text-to-speech Through Style Diffusion And Adversarial Training With Large Speech Language Models (2023)8.09
• ZS-MSTM: Zero-shot Style Transfer For Gesture Animation Driven By Text And Speech Using Adversarial Disentanglement Of Multimodal Style Encoding (2023)8.82
• Styletts-zs: Efficient High-quality Zero-shot Text-to-speech Synthesis With Distilled Time-varying Style Diffusion (2024)3.58
• Referee: Towards Reference-free Cross-speaker Style Transfer With Low-quality Data For Expressive Speech Synthesis (2021)7.16