Semi Supervised Phrase Localization In A Bidirectional Caption-image Retrieval Framework

Abstract

We introduce a novel deep neural network architecture that links visual regions to corresponding textual segments including phrases and words. To accomplish this task, our architecture makes use of the rich semantic information available in a joint embedding space of multi-modal data. From this joint embedding space, we extract the associative localization maps that develop naturally, without explicitly providing supervision during training for the localization task. The joint space is learned using a bidirectional ranking objective that is optimized using a \(N\)-Pair loss formulation. This training mechanism demonstrates the idea that localization information is learned inherently while optimizing a Bidirectional Retrieval objective. The model's retrieval and localization performance is evaluated on MSCOCO and Flickr30K Entities datasets. This architecture outperforms the state of the art results in the semi-supervised phrase localization setting.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydas2019semi

Related papers

• Align2ground: Weakly Supervised Phrase Grounding Guided By Image-caption Alignment (2019)13.93
• Finding Beans In Burgers: Deep Semantic-visual Embedding With Localization (2018)13.84
• Jointly Discovering Visual Objects And Spoken Words From Raw Sensory Input (2018)14.27
• Towards Retrieval-augmented Architectures For Image Captioning (2024)9.41
• Learning To Embed Semantic Similarity For Joint Image-text Retrieval (2022)7.50
• Constructing Phrase-level Semantic Labels To Form Multi-grained Supervision For Image-text Retrieval (2021)8.09
• Webly Supervised Joint Embedding For Cross-modal Image-text Retrieval (2018)13.17
• Ur2kid: Unifying Retrieval, Keypoint Detection, And Keypoint Description Without Local Correspondence Supervision (2020)0.00