Align2ground: Weakly Supervised Phrase Grounding Guided By Image-caption Alignment

Abstract

We address the problem of grounding free-form textual phrases by using weak supervision from image-caption pairs. We propose a novel end-to-end model that uses caption-to-image retrieval as a `downstream' task to guide the process of phrase localization. Our method, as a first step, infers the latent correspondences between regions-of-interest (RoIs) and phrases in the caption and creates a discriminative image representation using these matched RoIs. In a subsequent step, this (learned) representation is aligned with the caption. Our key contribution lies in building this `caption-conditioned' image encoding which tightly couples both the tasks and allows the weak supervision to effectively guide visual grounding. We provide an extensive empirical and qualitative analysis to investigate the different components of our proposed model and compare it with competitive baselines. For phrase localization, we report an improvement of 4.9% (absolute) over the prior state-of-the-art on the Vis

Stats

• citations71
• S2 citations—
• github stars0
• HF likes0
• heat score13.93
• arxiv keydatta2019align2ground

Related papers

• Semi Supervised Phrase Localization In A Bidirectional Caption-image Retrieval Framework (2019)0.00
• Learning Visually Grounded Sentence Representations (2017)7.81
• Grounding Language Models To Images For Multimodal Inputs And Outputs (2023)0.00
• Linear Alignment Of Vision-language Models For Image Captioning (2023)0.00
• Distinctive Image Captioning: Leveraging Ground Truth Captions In CLIP Guided Reinforcement Learning (2024)4.52
• Compositional Image-text Matching And Retrieval By Grounding Entities (2025)0.60
• Constructing Phrase-level Semantic Labels To Form Multi-grained Supervision For Image-text Retrieval (2021)8.09
• GOAL: Global-local Object Alignment Learning (2025)2.26