Transcription-enriched Joint Embeddings For Spoken Descriptions Of Images And Videos

Abstract

In this work, we propose an effective approach for training unique embedding representations by combining three simultaneous modalities: image and spoken and textual narratives. The proposed methodology departs from a baseline system that spawns a embedding space trained with only spoken narratives and image cues. Our experiments on the EPIC-Kitchen and Places Audio Caption datasets show that introducing the human-generated textual transcriptions of the spoken narratives helps to the training procedure yielding to get better embedding representations. The triad speech, image and words allows for a better estimate of the point embedding and show an improving of the performance within tasks like image and speech retrieval, even when text third modality, text, is not present in the task.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyoriol2020transcription

Related papers

• Webly Supervised Joint Embedding For Cross-modal Image-text Retrieval (2018)13.17
• Symbolic Inductive Bias For Visually Grounded Learning Of Spoken Language (2018)5.24
• Learning Joint Representations Of Videos And Sentences With Web Image Search (2016)12.93
• Fine-grained Action Retrieval Through Multiple Parts-of-speech Embeddings (2019)15.62
• Learnable Pins: Cross-modal Embeddings For Person Identity (2018)15.22
• Learning TFIDF Enhanced Joint Embedding For Recipe-image Cross-modal Retrieval Service (2021)10.85
• Learning Language-visual Embedding For Movie Understanding With Natural-language (2016)0.00
• An Analysis Of Semantically-aligned Speech-text Embeddings (2022)7.81