Transcription-enriched Joint Embeddings For Spoken Descriptions Of Images And Videos

Abstract

In this work, we propose an effective approach for training unique embedding representations by combining three simultaneous modalities: image and spoken and textual narratives. The proposed methodology departs from a baseline system that spawns a embedding space trained with only spoken narratives and image cues. Our experiments on the EPIC-Kitchen and Places Audio Caption datasets show that introducing the human-generated textual transcriptions of the spoken narratives helps to the training procedure yielding to get better embedding representations. The triad speech, image and words allows for a better estimate of the point embedding and show an improving of the performance within tasks like image and speech retrieval, even when text third modality, text, is not present in the task.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyoriol2020transcription

Related papers

• Large-scale Representation Learning From Visually Grounded Untranscribed Speech (2019)10.48
• Symbolic Inductive Bias For Visually Grounded Learning Of Spoken Language (2018)5.24
• Connecting The Dots Between Audio And Text Without Parallel Data Through Visual Knowledge Transfer (2021)8.09
• Contextual Joint Factor Acoustic Embeddings (2019)2.26
• Learning Speech Representations From Raw Audio By Joint Audiovisual Self-supervision (2020)0.00
• A Better Use Of Audio-visual Cues: Dense Video Captioning With Bi-modal Transformer (2020)10.61
• Direct Multimodal Few-shot Learning Of Speech And Images (2020)5.84
• Joint Learning Of Word And Label Embeddings For Sequence Labelling In Spoken Language Understanding (2019)3.58