Multi-grained Spatio-temporal Modeling For Lip-reading

Abstract

Lip-reading aims to recognize speech content from videos via visual analysis of speakers' lip movements. This is a challenging task due to the existence of homophemes-words which involve identical or highly similar lip movements, as well as diverse lip appearances and motion patterns among the speakers. To address these challenges, we propose a novel lip-reading model which captures not only the nuance between words but also styles of different speakers, by a multi-grained spatio-temporal modeling of the speaking process. Specifically, we first extract both frame-level fine-grained features and short-term medium-grained features by the visual front-end, which are then combined to obtain discriminative representations for words with similar phonemes. Next, a bidirectional ConvLSTM augmented with temporal attention aggregates spatio-temporal information in the entire input sequence, which is expected to be able to capture the coarse-gained patterns of each word and robust to various cond

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2019multi

Related papers

• Spatio-temporal Attention Mechanism And Knowledge Distillation For Lip Reading (2021)0.00
• Lipper: Synthesizing Thy Speech Using Multi-view Lipreading (2019)10.61
• Lipformer: Learning To Lipread Unseen Speakers Based On Visual-landmark Transformers (2023)11.49
• Simullr: Simultaneous Lip Reading Transducer With Attention-guided Adaptive Memory (2021)8.09
• Lipsound2: Self-supervised Pre-training For Lip-to-speech Reconstruction And Lip Reading (2021)11.39
• Learning Speaker-invariant Visual Features For Lipreading (2025)0.00
• Learning Separable Hidden Unit Contributions For Speaker-adaptive Lip-reading (2023)0.00
• Lipreading With 3D-2D-CNN BLSTM-HMM And Word-ctc Models (2019)0.00