Isochrony-aware Neural Machine Translation For Automatic Dubbing

Abstract

We introduce the task of isochrony-aware machine translation which aims at generating translations suitable for dubbing. Dubbing of a spoken sentence requires transferring the content as well as the speech-pause structure of the source into the target language to achieve audiovisual coherence. Practically, this implies correctly projecting pauses from the source to the target and ensuring that target speech segments have roughly the same duration of the corresponding source speech segments. In this work, we propose implicit and explicit modeling approaches to integrate isochrony information into neural machine translation. Experiments on English-German/French language pairs with automatic metrics show that the simplest of the considered approaches works best. Results are confirmed by human evaluations of translations and dubbed videos.

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keytam2021isochrony

Related papers

• Dubbing In Practice: A Large Scale Study Of Human Localization With Insights For Automatic Dubbing (2022)8.82
• Prosodic Phrase Alignment For Machine Dubbing (2019)8.60
• Isochronometer: A Simple And Effective Isochronic Translation Evaluation Metric (2024)0.60
• Videodubber: Machine Translation With Speech-aware Length Control For Video Dubbing (2022)8.82
• Improving Isochronous Machine Translation With Target Factors And Auxiliary Counters (2023)4.52
• Neural Dubber: Dubbing For Videos According To Scripts (2021)0.00
• Large-scale Multilingual Audio Visual Dubbing (2020)0.00
• Prosody-enhanced Acoustic Pre-training And Acoustic-disentangled Prosody Adapting For Movie Dubbing (2025)3.58