Facediffuser: Speech-driven 3D Facial Animation Synthesis Using Diffusion

Abstract

Speech-driven 3D facial animation synthesis has been a challenging task both in industry and research. Recent methods mostly focus on deterministic deep learning methods meaning that given a speech input, the output is always the same. However, in reality, the non-verbal facial cues that reside throughout the face are non-deterministic in nature. In addition, majority of the approaches focus on 3D vertex based datasets and methods that are compatible with existing facial animation pipelines with rigged characters is scarce. To eliminate these issues, we present FaceDiffuser, a non-deterministic deep learning model to generate speech-driven facial animations that is trained with both 3D vertex and blendshape based datasets. Our method is based on the diffusion technique and uses the pre-trained large speech representation model HuBERT to encode the audio input. To the best of our knowledge, we are the first to employ the diffusion method for the task of speech-driven 3D facial animation

Stats

• citations68
• S2 citations—
• github stars0
• HF likes0
• heat score13.79
• arxiv keystan2023facediffuser

Related papers

• Diffspeaker: Speech-driven 3D Facial Animation With Diffusion Transformer (2024)5.24
• Df-3dface: One-to-many Speech Synchronized 3D Face Animation With Diffusion (2023)0.00
• Said: Speech-driven Blendshape Facial Animation With Diffusion (2023)0.00
• Diffusiontalker: Efficient And Compact Speech-driven 3D Talking Head Via Personalizer-guided Distillation (2025)5.05
• Ksdiff: Keyframe-augmented Speech-aware Dual-path Diffusion For Facial Animation (2025)0.00
• Diffmotion: Speech-driven Gesture Synthesis Using Denoising Diffusion Model (2023)9.59
• Diffsheg: A Diffusion-based Approach For Real-time Speech-driven Holistic 3D Expression And Gesture Generation (2024)0.00
• Diffuse Or Confuse: A Diffusion Deepfake Speech Dataset (2024)5.24