FLOAT: Generative Motion Latent Flow Matching For Audio-driven Talking Portrait

Abstract

With the rapid advancement of diffusion-based generative models, portrait image animation has achieved remarkable results. However, it still faces challenges in temporally consistent video generation and fast sampling due to its iterative sampling nature. This paper presents FLOAT, an audio-driven talking portrait video generation method based on flow matching generative model. Instead of a pixel-based latent space, we take advantage of a learned orthogonal motion latent space, enabling efficient generation and editing of temporally consistent motion. To achieve this, we introduce a transformer-based vector field predictor with an effective frame-wise conditioning mechanism. Additionally, our method supports speech-driven emotion enhancement, enabling a natural incorporation of expressive motions. Extensive experiments demonstrate that our method outperforms state-of-the-art audio-driven talking portrait methods in terms of visual quality, motion fidelity, and efficiency.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyki2024float

Related papers

• See The Speaker: Crafting High-resolution Talking Faces From Speech With Prior Guidance And Region Refinement (2025)0.00
• Hallo3: Highly Dynamic And Realistic Portrait Image Animation With Video Diffusion Transformer (2024)8.35
• Dreamfoley: Scalable Vlms For High-fidelity Video-to-audio Generation (2025)0.00
• Real-time Streamable Generative Speech Restoration With Flow Matching (2025)0.00
• Fluentlip: A Phonemes-based Two-stage Approach For Audio-driven Lip Synthesis With Optical Flow Consistency (2025)0.00
• Flashaudio: Rectified Flows For Fast And High-fidelity Text-to-audio Generation (2024)5.13
• Voiceflow: Efficient Text-to-speech With Rectified Flow Matching (2023)0.00
• Freetalker: Controllable Speech And Text-driven Gesture Generation Based On Diffusion Models For Enhanced Speaker Naturalness (2024)9.59