Order Embeddings And Character-level Convolutions For Multimodal Alignment

Abstract

With the novel and fast advances in the area of deep neural networks, several challenging image-based tasks have been recently approached by researchers in pattern recognition and computer vision. In this paper, we address one of these tasks, which is to match image content with natural language descriptions, sometimes referred as multimodal content retrieval. Such a task is particularly challenging considering that we must find a semantic correspondence between captions and the respective image, a challenge for both computer vision and natural language processing areas. For such, we propose a novel multimodal approach based solely on convolutional neural networks for aligning images with their captions by directly convolving raw characters. Our proposed character-based textual embeddings allow the replacement of both word-embeddings and recurrent neural networks for text understanding, saving processing time and requiring fewer learnable parameters. Our method is based on the idea of

Stats

• citations15
• S2 citations—
• github stars0
• HF likes0
• heat score9.03
• arxiv keywehrmann2017order

Related papers

• Aligning Multilingual Word Embeddings For Cross-modal Retrieval Task (2019)2.26
• Deep Multimodal Image-text Embeddings For Automatic Cross-media Retrieval (2020)0.00
• Objembed: Towards Universal Multimodal Object Embeddings (2026)0.00
• MATE: Meet At The Embedding -- Connecting Images With Long Texts (2024)5.24
• Multimodal Representation Alignment For Cross-modal Information Retrieval (2025)0.00
• Image Search Using Multilingual Texts: A Cross-modal Learning Approach Between Image And Text (2019)0.00
• Deep Unified Multimodal Embeddings For Understanding Both Content And Users In Social Media Networks (2019)0.00
• Dynamic Visual Semantic Sub-embeddings And Fast Re-ranking (2023)0.00