Learning Modality-invariant Representations For Speech And Images

Abstract

In this paper, we explore the unsupervised learning of a semantic embedding space for co-occurring sensory inputs. Specifically, we focus on the task of learning a semantic vector space for both spoken and handwritten digits using the TIDIGITs and MNIST datasets. Current techniques encode image and audio/textual inputs directly to semantic embeddings. In contrast, our technique maps an input to the mean and log variance vectors of a diagonal Gaussian from which sample semantic embeddings are drawn. In addition to encouraging semantic similarity between co-occurring inputs,our loss function includes a regularization term borrowed from variational autoencoders (VAEs) which drives the posterior distributions over embeddings to be unit Gaussian. We can use this regularization term to filter out modality information while preserving semantic information. We speculate this technique may be more broadly applicable to other areas of cross-modality/domain information retrieval and transfer lear

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keyleidal2017learning

Related papers

• Multimodal One-shot Learning Of Speech And Images (2018)9.03
• Learning Speaker Representations With Mutual Information (2018)11.76
• Vision As An Interlingua: Learning Multilingual Semantic Embeddings Of Untranscribed Speech (2018)10.74
• A Multimodal Dynamical Variational Autoencoder For Audiovisual Speech Representation Learning (2023)2.26
• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Cross-modal Audio-visual Co-learning For Text-independent Speaker Verification (2023)9.23
• Tie Your Embeddings Down: Cross-modal Latent Spaces For End-to-end Spoken Language Understanding (2020)9.03
• Cross-modal Denoising: A Novel Training Paradigm For Enhancing Speech-image Retrieval (2024)0.00