Direct Speech Translation For Automatic Subtitling

Abstract

Automatic subtitling is the task of automatically translating the speech of audiovisual content into short pieces of timed text, i.e. subtitles and their corresponding timestamps. The generated subtitles need to conform to space and time requirements, while being synchronised with the speech and segmented in a way that facilitates comprehension. Given its considerable complexity, the task has so far been addressed through a pipeline of components that separately deal with transcribing, translating, and segmenting text into subtitles, as well as predicting timestamps. In this paper, we propose the first direct ST model for automatic subtitling that generates subtitles in the target language along with their timestamps with a single model. Our experiments on 7 language pairs show that our approach outperforms a cascade system in the same data condition, also being competitive with production tools on both in-domain and newly-released out-domain benchmarks covering new scenarios.

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keypapi2022direct

Related papers

• Dodging The Data Bottleneck: Automatic Subtitling With Automatically Segmented ST Corpora (2022)2.26
• Between Flexibility And Consistency: Joint Generation Of Captions And Subtitles (2021)5.24
• Direct Models For Simultaneous Translation And Automatic Subtitling: FBK@IWSLT2023 (2023)2.26
• Learning To Jointly Transcribe And Subtitle For End-to-end Spontaneous Speech Recognition (2022)5.84
• Is 42 The Answer To Everything In Subtitling-oriented Speech Translation? (2020)7.81
• Leveraging Broadcast Media Subtitle Transcripts For Automatic Speech Recognition And Subtitling (2025)2.26
• Synchronous Speech Recognition And Speech-to-text Translation With Interactive Decoding (2019)10.48
• Multilingual End-to-end Speech Translation (2019)0.00