Watch Or Listen: Robust Audio-visual Speech Recognition With Visual Corruption Modeling And Reliability Scoring
2023 · Joanna Hong, Minsu Kim, Jeongsoo Choi, et al.
Abstract
This paper deals with Audio-Visual Speech Recognition (AVSR) under multimodal input corruption situations where audio inputs and visual inputs are both corrupted, which is not well addressed in previous research directions. Previous studies have focused on how to complement the corrupted audio inputs with the clean visual inputs with the assumption of the availability of clean visual inputs. However, in real life, clean visual inputs are not always accessible and can even be corrupted by occluded lip regions or noises. Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models. Then, we design multimodal input corruption modeling to develop robust AVSR models. Lastly, we propose a novel AVSR framework, namely Audio-Visual Reliability Scoring module (AV-RelScore), that is robust to the corrupted multimodal inputs. The AV-RelScore can determine which input modal
Authors
(none)
Tags
Stats
Related papers
- Robust End-to-end Deep Audiovisual Speech Recognition (2016)0.00
- MLCA-AVSR: Multi-layer Cross Attention Fusion Based Audio-visual Speech Recognition (2024)10.07
- Listening And Seeing Again: Generative Error Correction For Audio-visual Speech Recognition (2025)1.20
- XLAVS-R: Cross-lingual Audio-visual Speech Representation Learning For Noise-robust Speech Perception (2024)7.50
- Visual Context-driven Audio Feature Enhancement For Robust End-to-end Audio-visual Speech Recognition (2022)10.07
- Multilingual Audio-visual Speech Recognition With Hybrid CTC/RNN-T Fast Conformer (2024)8.60
- Robust Audio-visual Speech Enhancement: Correcting Misassignments In Complex Environments With Advanced Post-processing (2024)0.00
- Cross-modal Global Interaction And Local Alignment For Audio-visual Speech Recognition (2023)7.50