Text-driven Talking Face Synthesis By Reprogramming Audio-driven Models

Abstract

In this paper, we present a method for reprogramming pre-trained audio-driven talking face synthesis models to operate in a text-driven manner. Consequently, we can easily generate face videos that articulate the provided textual sentences, eliminating the necessity of recording speech for each inference, as required in the audio-driven model. To this end, we propose to embed the input text into the learned audio latent space of the pre-trained audio-driven model, while preserving the face synthesis capability of the original pre-trained model. Specifically, we devise a Text-to-Audio Embedding Module (TAEM) which maps a given text input into the audio latent space by modeling pronunciation and duration characteristics. Furthermore, to consider the speaker characteristics in audio while using text inputs, TAEM is designed to accept a visual speaker embedding. The visual speaker embedding is derived from a single target face image and enables improved mapping of input text to the learned

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keychoi2023text

Related papers

• Transface: Unit-based Audio-visual Speech Synthesizer For Talking Head Translation (2023)7.16
• Audio2face: Generating Speech/face Animation From Single Audio With Attention-based Bidirectional LSTM Networks (2019)12.10
• Facetron: A Multi-speaker Face-to-speech Model Based On Cross-modal Latent Representations (2021)0.00
• Audiovisual Speech Synthesis Using Tacotron2 (2020)8.09
• See The Speaker: Crafting High-resolution Talking Faces From Speech With Prior Guidance And Region Refinement (2025)0.00
• Emogene: Audio-driven Emotional 3D Talking-head Generation (2024)2.26
• A Unified Compression Framework For Efficient Speech-driven Talking-face Generation (2023)0.00
• Text-to-video: A Two-stage Framework For Zero-shot Identity-agnostic Talking-head Generation (2023)1.69