Vid2speech: Speech Reconstruction From Silent Video

Abstract

Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.

Stats

• citations76
• S2 citations—
• github stars0
• HF likes0
• heat score14.15
• arxiv keyephrat2017vid2speech

Related papers

• Improved Speech Reconstruction From Silent Video (2017)13.34
• Video-driven Speech Reconstruction Using Generative Adversarial Networks (2019)11.39
• Lipvoicer: Generating Speech From Silent Videos Guided By Lip Reading (2023)3.89
• Let There Be Sound: Reconstructing High Quality Speech From Silent Videos (2023)6.34
• Lipsound2: Self-supervised Pre-training For Lip-to-speech Reconstruction And Lip Reading (2021)11.39
• Speech Reconstruction From Silent Tongue And Lip Articulation By Pseudo Target Generation And Domain Adversarial Training (2023)5.84
• Neural2speech: A Transfer Learning Framework For Neural-driven Speech Reconstruction (2023)5.24
• Audio-visual Video-to-speech Synthesis With Synthesized Input Audio (2023)0.00