Audio Input Generates Continuous Frames To Synthesize Facial Video Using Generative Adiversarial Networks

Abstract

This paper presents a simple method for speech videos generation based on audio: given a piece of audio, we can generate a video of the target face speaking this audio. We propose Generative Adversarial Networks (GAN) with cut speech audio input as condition and use Convolutional Gate Recurrent Unit (GRU) in generator and discriminator. Our model is trained by exploiting the short audio and the frames in this duration. For training, we cut the audio and extract the face in the corresponding frames. We designed a simple encoder and compare the generated frames using GAN with and without GRU. We use GRU for temporally coherent frames and the results show that short audio can produce relatively realistic output results.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhang2022audio

Related papers

• Video-driven Speech Reconstruction Using Generative Adversarial Networks (2019)11.39
• End-to-end Video-to-speech Synthesis Using Generative Adversarial Networks (2021)11.58
• Audio2face: Generating Speech/face Animation From Single Audio With Attention-based Bidirectional LSTM Networks (2019)12.10
• From Inference To Generation: End-to-end Fully Self-supervised Generation Of Human Face From Speech (2020)0.00
• Voice Command Generation Using Progressive Wavegans (2019)0.00
• High Fidelity Speech Synthesis With Adversarial Networks (2019)0.00
• Adversarial Audio Synthesis (2018)0.00
• Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks (2017)16.21