Vision As An Interlingua: Learning Multilingual Semantic Embeddings Of Untranscribed Speech

Abstract

In this paper, we explore the learning of neural network embeddings for natural images and speech waveforms describing the content of those images. These embeddings are learned directly from the waveforms without the use of linguistic transcriptions or conventional speech recognition technology. While prior work has investigated this setting in the monolingual case using English speech data, this work represents the first effort to apply these techniques to languages beyond English. Using spoken captions collected in English and Hindi, we show that the same model architecture can be successfully applied to both languages. Further, we demonstrate that training a multilingual model simultaneously on both languages offers improved performance over the monolingual models. Finally, we show that these models are capable of performing semantic cross-lingual speech-to-speech retrieval.

Stats

• citations26
• S2 citations—
• github stars0
• HF likes0
• heat score10.74
• arxiv keyharwath2018vision

Related papers

• Language Learning Using Speech To Image Retrieval (2019)9.41
• Hindi As A Second Language: Improving Visually Grounded Speech With Semantically Similar Samples (2023)6.77
• Multilingual Acoustic Word Embedding Models For Processing Zero-resource Languages (2020)8.09
• Improved Acoustic Word Embeddings For Zero-resource Languages Using Multilingual Transfer (2020)7.81
• Neuralmultiling: A Novel Neural Architecture Search For Smartphone Based Multilingual Speaker Verification (2024)0.00
• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Intuitive Multilingual Audio-visual Speech Recognition With A Single-trained Model (2023)5.24
• Large-scale Representation Learning From Visually Grounded Untranscribed Speech (2019)10.48