CAMP: Cross-modal Adaptive Message Passing For Text-image Retrieval

Abstract

Text-image cross-modal retrieval is a challenging task in the field of language and vision. Most previous approaches independently embed images and sentences into a joint embedding space and compare their similarities. However, previous approaches rarely explore the interactions between images and sentences before calculating similarities in the joint space. Intuitively, when matching between images and sentences, human beings would alternatively attend to regions in images and words in sentences, and select the most salient information considering the interaction between both modalities. In this paper, we propose Cross-modal Adaptive Message Passing (CAMP), which adaptively controls the information flow for message passing across modalities. Our approach not only takes comprehensive and fine-grained cross-modal interactions into account, but also properly handles negative pairs and irrelevant information with an adaptive gating scheme. Moreover, instead of conventional joint embedding

Stats

• citations281
• S2 citations—
• github stars0
• HF likes0
• heat score18.38
• arxiv keywang2019camp

Related papers

• Deep Multimodal Image-text Embeddings For Automatic Cross-media Retrieval (2020)0.00
• COTS: Collaborative Two-stream Vision-language Pre-training Model For Cross-modal Retrieval (2022)13.60
• Revisiting Cross Modal Retrieval (2018)0.00
• Image Search Using Multilingual Texts: A Cross-modal Learning Approach Between Image And Text (2019)0.00
• Look, Imagine And Match: Improving Textual-visual Cross-modal Retrieval With Generative Models (2017)18.52
• IMRAM: Iterative Matching With Recurrent Attention Memory For Cross-modal Image-text Retrieval (2020)19.22
• Cross-modal Coherence For Text-to-image Retrieval (2021)6.77
• Intra-modal Constraint Loss For Image-text Retrieval (2022)8.33