More Than Words: In-the-wild Visually-driven Prosody For Text-to-speech
2021 · Michael Hassid, Michelle Tadmor Ramanovich, Brendan Shillingford, et al.
Abstract
In this paper we present VDTTS, a Visually-Driven Text-to-Speech model. Motivated by dubbing, VDTTS takes advantage of video frames as an additional input alongside text, and generates speech that matches the video signal. We demonstrate how this allows VDTTS to, unlike plain TTS models, generate speech that not only has prosodic variations like natural pauses and pitch, but is also synchronized to the input video. Experimentally, we show our model produces well-synchronized outputs, approaching the video-speech synchronization quality of the ground-truth, on several challenging benchmarks including "in-the-wild" content from VoxCeleb2. Supplementary demo videos demonstrating video-speech synchronization, robustness to speaker ID swapping, and prosody, presented at the project page.
Authors
(none)
Tags
Stats
Related papers
- Visualtts: TTS With Accurate Lip-speech Synchronization For Automatic Voice Over (2021)9.41
- Mechanisms Of Multimodal Synchronization: Insights From Decoder-based Video-text-to-speech Synthesis (2024)0.00
- Vit-tts: Visual Text-to-speech With Scalable Diffusion Transformer (2023)7.16
- Diffv2s: Diffusion-based Video-to-speech Synthesis With Vision-guided Speaker Embedding (2023)8.82
- Voicecraft-dub: Automated Video Dubbing With Neural Codec Language Models (2025)0.00
- Divise: Direct Visual-input Speech Synthesis Preserving Speaker Characteristics And Intelligibility (2025)5.58
- Talkverse: Democratizing Minute-long Audio-driven Video Generation (2025)0.00
- Learning To Dub Movies Via Hierarchical Prosody Models (2022)10.97