Cross-modal Semantic Enhanced Interaction For Image-sentence Retrieval

Abstract

Image-sentence retrieval has attracted extensive research attention in multimedia and computer vision due to its promising application. The key issue lies in jointly learning the visual and textual representation to accurately estimate their similarity. To this end, the mainstream schema adopts an object-word based attention to calculate their relevance scores and refine their interactive representations with the attention features, which, however, neglects the context of the object representation on the inter-object relationship that matches the predicates in sentences. In this paper, we propose a Cross-modal Semantic Enhanced Interaction method, termed CMSEI for image-sentence retrieval, which correlates the intra- and inter-modal semantics between objects and words. In particular, we first design the intra-modal spatial and semantic graphs based reasoning to enhance the semantic representations of objects guided by the explicit relationships of the objects' spatial positions and the

Stats

• citations43
• S2 citations—
• github stars0
• HF likes0
• heat score12.33
• arxiv keyge2022cross

Related papers

• Structured Multi-modal Feature Embedding And Alignment For Image-sentence Retrieval (2021)12.87
• Semantic-enhanced Modality-asymmetric Retrieval For Online E-commerce Search (2025)0.00
• Do Cross Modal Systems Leverage Semantic Relationships? (2019)7.16
• Preserving Semantic Neighborhoods For Robust Cross-modal Retrieval (2020)10.07
• CAMP: Cross-modal Adaptive Message Passing For Text-image Retrieval (2019)18.38
• A Novel Self-supervised Cross-modal Image Retrieval Method In Remote Sensing (2022)8.35
• Revisiting Cross Modal Retrieval (2018)0.00
• SAC: Semantic Attention Composition For Text-conditioned Image Retrieval (2020)11.49