Deep Latent Space Learning For Cross-modal Mapping Of Audio And Visual Signals

Abstract

We propose a novel deep training algorithm for joint representation of audio and visual information which consists of a single stream network (SSNet) coupled with a novel loss function to learn a shared deep latent space representation of multimodal information. The proposed framework characterizes the shared latent space by leveraging the class centers which helps to eliminate the need for pairwise or triplet supervision. We quantitatively and qualitatively evaluate the proposed approach on VoxCeleb, a benchmarks audio-visual dataset on a multitude of tasks including cross-modal verification, cross-modal matching, and cross-modal retrieval. State-of-the-art performance is achieved on cross-modal verification and matching while comparable results are observed on the remaining applications. Our experiments demonstrate the effectiveness of the technique for cross-modal biometric applications.

Stats

• citations41
• S2 citations—
• github stars0
• HF likes0
• heat score12.17
• arxiv keynawaz2019deep

Related papers

• Unsupervised Cross-modal Audio Representation Learning From Unstructured Multilingual Text (2020)2.26
• See, Hear, And Read: Deep Aligned Representations (2017)0.00
• Perfect Match: Improved Cross-modal Embeddings For Audio-visual Synchronisation (2018)14.19
• Deep Triplet Neural Networks With Cluster-cca For Audio-visual Cross-modal Retrieval (2019)12.61
• Coordinated Joint Multimodal Embeddings For Generalized Audio-visual Zeroshot Classification And Retrieval Of Videos (2019)12.54
• VMCML: Video And Music Matching Via Cross-modality Lifting (2023)2.26
• Cross-modal Embeddings For Video And Audio Retrieval (2018)11.08
• Cross-modal Discrete Representation Learning (2021)10.61