Self-supervised Learning From Web Data For Multimodal Retrieval

Abstract

Self-Supervised learning from multimodal image and text data allows deep neural networks to learn powerful features with no need of human annotated data. Web and Social Media platforms provide a virtually unlimited amount of this multimodal data. In this work we propose to exploit this free available data to learn a multimodal image and text embedding, aiming to leverage the semantic knowledge learnt in the text domain and transfer it to a visual model for semantic image retrieval. We demonstrate that the proposed pipeline can learn from images with associated textwithout supervision and analyze the semantic structure of the learnt joint image and text embedding space. We perform a thorough analysis and performance comparison of five different state of the art text embeddings in three different benchmarks. We show that the embeddings learnt with Web and Social Media data have competitive performances over supervised methods in the text based image retrieval task, and we clearly outperf

Stats

• citations11
• S2 citations—
• github stars0
• HF likes0
• heat score8.09
• arxiv keygomez2019self

Related papers

• Learning To Learn From Web Data Through Deep Semantic Embeddings (2018)9.03
• Webly Supervised Joint Embedding For Cross-modal Image-text Retrieval (2018)13.17
• Multimodal Clustering Networks For Self-supervised Learning From Unlabeled Videos (2021)13.28
• Deep Multimodal Image-text Embeddings For Automatic Cross-media Retrieval (2020)0.00
• Self-supervised Visual Representations For Cross-modal Retrieval (2019)7.50
• Learning Robust Visual-semantic Embeddings (2017)15.22
• Semi-supervised Cross-modal Retrieval With Label Prediction (2018)11.29
• Preserving Semantic Neighborhoods For Robust Cross-modal Retrieval (2020)10.07