Learning To Dub Movies Via Hierarchical Prosody Models
2022 · Gaoxiang Cong, Liang Li, Yuankai Qi, et al.
Abstract
Given a piece of text, a video clip and a reference audio, the movie dubbing (also known as visual voice clone V2C) task aims to generate speeches that match the speaker's emotion presented in the video using the desired speaker voice as reference. V2C is more challenging than conventional text-to-speech tasks as it additionally requires the generated speech to exactly match the varying emotions and speaking speed presented in the video. Unlike previous works, we propose a novel movie dubbing architecture to tackle these problems via hierarchical prosody modelling, which bridges the visual information to corresponding speech prosody from three aspects: lip, face, and scene. Specifically, we align lip movement to the speech duration, and convey facial expression to speech energy and pitch via attention mechanism based on valence and arousal representations inspired by recent psychology findings. Moreover, we design an emotion booster to capture the atmosphere from global video scenes. A
Authors
(none)
Tags
Stats
Related papers
- Emodubber: Towards High Quality And Emotion Controllable Movie Dubbing (2024)4.52
- Prosody-enhanced Acoustic Pre-training And Acoustic-disentangled Prosody Adapting For Movie Dubbing (2025)3.58
- Neural Dubber: Dubbing For Videos According To Scripts (2021)0.00
- Voicecraft-dub: Automated Video Dubbing With Neural Codec Language Models (2025)0.00
- Mcdubber: Multimodal Context-aware Expressive Video Dubbing (2024)5.91
- Towards Expressive Video Dubbing With Multiscale Multimodal Context Interaction (2024)4.52
- Large-scale Multilingual Audio Visual Dubbing (2020)0.00
- From Faces To Voices: Learning Hierarchical Representations For High-quality Video-to-speech (2025)0.00