Cross-modal Global Interaction And Local Alignment For Audio-visual Speech Recognition

Abstract

Audio-visual speech recognition (AVSR) research has gained a great success recently by improving the noise-robustness of audio-only automatic speech recognition (ASR) with noise-invariant visual information. However, most existing AVSR approaches simply fuse the audio and visual features by concatenation, without explicit interactions to capture the deep correlations between them, which results in sub-optimal multimodal representations for downstream speech recognition task. In this paper, we propose a cross-modal global interaction and local alignment (GILA) approach for AVSR, which captures the deep audio-visual (A-V) correlations from both global and local perspectives. Specifically, we design a global interaction model to capture the A-V complementary relationship on modality level, as well as a local alignment approach to model the A-V temporal consistency on frame level. Such a holistic view of cross-modal correlations enable better multimodal representations for AVSR. Experiment

Stats

• citations9
• S2 citations—
• github stars0
• HF likes0
• heat score7.50
• arxiv keyhu2023cross

Related papers

• Alignvsr: Audio-visual Cross-modal Alignment For Visual Speech Recognition (2024)0.00
• MLCA-AVSR: Multi-layer Cross Attention Fusion Based Audio-visual Speech Recognition (2024)10.07
• Hourglass-avsr: Down-up Sampling-based Computational Efficiency Model For Audio-visual Speech Recognition (2023)0.00
• How To Teach Dnns To Pay Attention To The Visual Modality In Speech Recognition (2020)10.97
• Cross-modal Audio-visual Co-learning For Text-independent Speaker Verification (2023)9.23
• Evaluation Of Audio-visual Alignments In Visually Grounded Speech Models (2021)5.84
• Listening And Seeing Again: Generative Error Correction For Audio-visual Speech Recognition (2025)1.20
• Improving Audio-visual Speech Recognition By Lip-subword Correlation Based Visual Pre-training And Cross-modal Fusion Encoder (2023)6.34