Isochrony-controlled Speech-to-text Translation: A Study On Translating From Sino-tibetan To Indo-european Languages

Abstract

End-to-end speech translation (ST), which translates source language speech directly into target language text, has garnered significant attention in recent years. Many ST applications require strict length control to ensure that the translation duration matches the length of the source audio, including both speech and pause segments. Previous methods often controlled the number of words or characters generated by the Machine Translation model to approximate the source sentence's length without considering the isochrony of pauses and speech segments, as duration can vary between languages. To address this, we present improvements to the duration alignment component of our sequence-to-sequence ST model. Our method controls translation length by predicting the duration of speech and pauses in conjunction with the translation process. This is achieved by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the transl

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyousefi2024isochrony

Related papers

• Synchronous Speech Recognition And Speech-to-text Translation With Interactive Decoding (2019)10.48
• Phonology-guided Speech-to-speech Translation For African Languages (2024)2.26
• Long-form End-to-end Speech Translation Via Latent Alignment Segmentation (2023)0.00
• Multilingual End-to-end Speech Translation (2019)0.00
• Leveraging Weakly Supervised Data To Improve End-to-end Speech-to-text Translation (2018)13.05
• Joint Training And Decoding For Multilingual End-to-end Simultaneous Speech Translation (2025)0.95
• From Start To Finish: Latency Reduction Strategies For Incremental Speech Synthesis In Simultaneous Speech-to-speech Translation (2021)2.26
• End-to-end Speech-to-text Translation: A Survey (2023)0.00