Facetron: A Multi-speaker Face-to-speech Model Based On Cross-modal Latent Representations
2021 · Se-Yun Um, Jihyun Kim, Jihyun Lee, et al.
Abstract
In this paper, we propose a multi-speaker face-to-speech waveform generation model that also works for unseen speaker conditions. Using a generative adversarial network (GAN) with linguistic and speaker characteristic features as auxiliary conditions, our method directly converts face images into speech waveforms under an end-to-end training framework. The linguistic features are extracted from lip movements using a lip-reading model, and the speaker characteristic features are predicted from face images using cross-modal learning with a pre-trained acoustic model. Since these two features are uncorrelated and controlled independently, we can flexibly synthesize speech waveforms whose speaker characteristics vary depending on the input face images. We show the superiority of our proposed model over conventional methods in terms of objective and subjective evaluation results. Specifically, we evaluate the performances of linguistic features by measuring their accuracy on an automatic sp
Authors
(none)
Tags
Stats
Related papers
- From Inference To Generation: End-to-end Fully Self-supervised Generation Of Human Face From Speech (2020)0.00
- Audio2face: Generating Speech/face Animation From Single Audio With Attention-based Bidirectional LSTM Networks (2019)12.10
- Reconstructing Faces From Voices (2019)0.00
- Ganspeech: Adversarial Training For High-fidelity Multi-speaker Speech Synthesis (2021)10.07
- From Faces To Voices: Learning Hierarchical Representations For High-quality Video-to-speech (2025)0.00
- Text-driven Talking Face Synthesis By Reprogramming Audio-driven Models (2023)2.26
- Multi-spectrogan: High-diversity And High-fidelity Spectrogram Generation With Adversarial Style Combination For Speech Synthesis (2020)0.00
- Audio Input Generates Continuous Frames To Synthesize Facial Video Using Generative Adiversarial Networks (2022)0.00