Stacked Cross Attention For Image-text Matching

Abstract

In this paper, we study the problem of image-text matching. Inferring the latent semantic alignment between objects or other salient stuff (e.g. snow, sky, lawn) and the corresponding words in sentences allows to capture fine-grained interplay between vision and language, and makes image-text matching more interpretable. Prior work either simply aggregates the similarity of all possible pairs of regions and words without attending differentially to more and less important words or regions, or uses a multi-step attentional process to capture limited number of semantic alignments which is less interpretable. In this paper, we present Stacked Cross Attention to discover the full latent alignments using both image regions and words in a sentence as context and infer image-text similarity. Our approach achieves the state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K, our approach outperforms the current best methods by 22.1% relatively in text retrieval from image q

Stats

• citations923
• S2 citations—
• github stars580
• HF likes0
• heat score27.77
• arxiv keylee2018stacked

Related papers

• Visual Semantic Reasoning For Image-text Matching (2019)25.23
• A New Fine-grained Alignment Method For Image-text Matching (2023)0.00
• ALADIN: Distilling Fine-grained Alignment Scores For Efficient Image-text Matching And Retrieval (2022)14.00
• MVAM: Multi-view Attention Method For Fine-grained Image-text Matching (2024)0.00
• IMRAM: Iterative Matching With Recurrent Attention Memory For Cross-modal Image-text Retrieval (2020)19.22
• Fine-grained Visual Textual Alignment For Cross-modal Retrieval Using Transformer Encoders (2020)19.48
• Deep Multimodal Image-text Embeddings For Automatic Cross-media Retrieval (2020)0.00
• Towards Fast And Accurate Image-text Retrieval With Self-supervised Fine-grained Alignment (2023)11.99