Large-scale Multilingual Audio Visual Dubbing

Abstract

We describe a system for large-scale audiovisual translation and dubbing, which translates videos from one language to another. The source language's speech content is transcribed to text, translated, and automatically synthesized into target language speech using the original speaker's voice. The visual content is translated by synthesizing lip movements for the speaker to match the translated audio, creating a seamless audiovisual experience in the target language. The audio and visual translation subsystems each contain a large-scale generic synthesis model trained on thousands of hours of data in the corresponding domain. These generic models are fine-tuned to a specific speaker before translation, either using an auxiliary corpus of data from the target speaker, or using the video to be translated itself as the input to the fine-tuning process. This report gives an architectural overview of the full system, as well as an in-depth discussion of the video dubbing component. The role

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyang2020large

Related papers

• Dubbing In Practice: A Large Scale Study Of Human Localization With Insights For Automatic Dubbing (2022)8.82
• Voicecraft-dub: Automated Video Dubbing With Neural Codec Language Models (2025)0.00
• Dubwise: Video-guided Speech Duration Control In Multimodal Llm-based Text-to-speech For Dubbing (2024)3.58
• Learning To Dub Movies Via Hierarchical Prosody Models (2022)10.97
• Neural Dubber: Dubbing For Videos According To Scripts (2021)0.00
• ANIM-400K: A Large-scale Dataset For Automated End-to-end Dubbing Of Video (2024)8.65
• Videodubber: Machine Translation With Speech-aware Length Control For Video Dubbing (2022)8.82
• Transface: Unit-based Audio-visual Speech Synthesizer For Talking Head Translation (2023)7.16