Harnessing Gans For Zero-shot Learning Of New Classes In Visual Speech Recognition

Abstract

Visual Speech Recognition (VSR) is the process of recognizing or interpreting speech by watching the lip movements of the speaker. Recent machine learning based approaches model VSR as a classification problem; however, the scarcity of training data leads to error-prone systems with very low accuracies in predicting unseen classes. To solve this problem, we present a novel approach to zero-shot learning by generating new classes using Generative Adversarial Networks (GANs), and show how the addition of unseen class samples increases the accuracy of a VSR system by a significant margin of 27% and allows it to handle speaker-independent out-of-vocabulary phrases. We also show that our models are language agnostic and therefore capable of seamlessly generating, using English training data, videos for a new language (Hindi). To the best of our knowledge, this is the first work to show empirical evidence of the use of GANs for generating training samples of unseen classes in the domain of V

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keykumar2019harnessing

Related papers

• SLMGAN: Exploiting Speech Language Model Representations For Unsupervised Zero-shot Voice Conversion In Gans (2023)0.00
• Video-driven Speech Reconstruction Using Generative Adversarial Networks (2019)11.39
• VSEGAN: Visual Speech Enhancement Generative Adversarial Network (2021)8.60
• Stargan-zsvc: Towards Zero-shot Voice Conversion In Low-resource Contexts (2021)3.58
• GAZEV: Gan-based Zero-shot Voice Conversion Over Non-parallel Speech Corpus (2020)8.60
• Analysis By Adversarial Synthesis -- A Novel Approach For Speech Vocoding (2019)3.58
• La-voce: Low-snr Audio-visual Speech Enhancement Using Neural Vocoders (2022)0.00
• On Enhancing Speech Emotion Recognition Using Generative Adversarial Networks (2018)12.33