Pseudo-convolutional Policy Gradient For Sequence-to-sequence Lip-reading
2020 · Mingshuang Luo, Shuang Yang, Shiguang Shan, et al.
Abstract
Lip-reading aims to infer the speech content from the lip movement sequence and can be seen as a typical sequence-to-sequence (seq2seq) problem which translates the input image sequence of lip movements to the text sequence of the speech content. However, the traditional learning process of seq2seq models always suffers from two problems: the exposure bias resulted from the strategy of "teacher-forcing", and the inconsistency between the discriminative optimization target (usually the cross-entropy loss) and the final evaluation metric (usually the character/word error rate). In this paper, we propose a novel pseudo-convolutional policy gradient (PCPG) based method to address these two problems. On the one hand, we introduce the evaluation metric (refers to the character error rate in this paper) as a form of reward to optimize the model together with the original discriminative target. On the other hand, inspired by the local perception property of convolutional operation, we perform
Authors
(none)
Tags
Stats
Related papers
- Multi-grained Spatio-temporal Modeling For Lip-reading (2019)0.00
- Learning Separable Hidden Unit Contributions For Speaker-adaptive Lip-reading (2023)0.00
- Lipvoicer: Generating Speech From Silent Videos Guided By Lip Reading (2023)3.89
- Lipreading Using Temporal Convolutional Networks (2020)17.61
- Simullr: Simultaneous Lip Reading Transducer With Attention-guided Adaptive Memory (2021)8.09
- Lipper: Synthesizing Thy Speech Using Multi-view Lipreading (2019)10.61
- Learning Online Alignments With Continuous Rewards Policy Gradient (2016)8.60
- Lipsound2: Self-supervised Pre-training For Lip-to-speech Reconstruction And Lip Reading (2021)11.39