Real-time Lip Sync For Live 2D Animation

Abstract

The emergence of commercial tools for real-time performance-based 2D animation has enabled 2D characters to appear on live broadcasts and streaming platforms. A key requirement for live animation is fast and accurate lip sync that allows characters to respond naturally to other actors or the audience through the voice of a human performer. In this work, we present a deep learning based interactive system that automatically generates live lip sync for layered 2D characters using a Long Short Term Memory (LSTM) model. Our system takes streaming audio as input and produces viseme sequences with less than 200ms of latency (including processing time). Our contributions include specific design decisions for our feature definition and LSTM configuration that provide a small but useful amount of lookahead to produce accurate lip sync. We also describe a data augmentation procedure that allows us to achieve good results with a very small amount of hand-animated training data (13-20 minutes). Ex

Stats

• citations0
• S2 citations—
• github stars148
• HF likes0
• heat score4.35
• arxiv keyaneja2019real

Related papers

• Audio2face: Generating Speech/face Animation From Single Audio With Attention-based Bidirectional LSTM Networks (2019)12.10
• Lpips-attnwav2lip: Generic Audio-driven Lip Synchronization For Talking Head Generation In The Wild (2026)12.65
• Fluentlip: A Phonemes-based Two-stage Approach For Audio-driven Lip Synthesis With Optical Flow Consistency (2025)0.00
• Visualtts: TTS With Accurate Lip-speech Synchronization For Automatic Voice Over (2021)9.41
• Improving Lip-synchrony In Direct Audio-visual Speech-to-speech Translation (2024)0.00
• Keyface: Expressive Audio-driven Facial Animation For Long Sequences Via Keyframe Interpolation (2025)4.52
• Transformer-s2a: Robust And Efficient Speech-to-animation (2021)8.35
• Style-preserving Lip Sync Via Audio-aware Style Reference (2024)0.00