Multi-modality Associative Bridging Through Memory: Speech Sound Recollected From Face Video

Abstract

In this paper, we introduce a novel audio-visual multi-modal bridging framework that can utilize both audio and visual information, even with uni-modal inputs. We exploit a memory network that stores source (i.e., visual) and target (i.e., audio) modal representations, where source modal representation is what we are given, and target modal representations are what we want to obtain from the memory network. We then construct an associative bridge between source and target memories that considers the interrelationship between the two memories. By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks. We apply the proposed framework to two tasks: lip reading and speech reconstruction from silent video. Through the proposed associative bridge and modality-specific memories, ea

Stats

• citations40
• S2 citations—
• github stars0
• HF likes0
• heat score12.10
• arxiv keykim2022multi

Related papers

• Multi-modal Multi-correlation Learning For Audio-visual Speech Separation (2022)5.84
• Cross-modal Audio-visual Co-learning For Text-independent Speaker Verification (2023)9.23
• Cross-modal Speaker Verification And Recognition: A Multilingual Perspective (2020)0.00
• Audio-visual Speech Separation Based On Joint Feature Representation With Cross-modal Attention (2022)0.00
• Chatbridge: Bridging Modalities With Large Language Model As A Language Catalyst (2023)0.00
• Unibrivl: Robust Universal Representation And Generation Of Audio Driven Diffusion Models (2023)2.26
• Attention-based Cross-modal Fusion For Audio-visual Voice Activity Detection In Musical Video Streams (2021)5.24
• Attentive Fusion Enhanced Audio-visual Encoding For Transformer Based Robust Speech Recognition (2020)0.00