Image Retrieval On Real-life Images With Pre-trained Vision-and-language Models

Abstract

We extend the task of composed image retrieval, where an input query consists of an image and short textual description of how to modify the image. Existing methods have only been applied to non-complex images within narrow domains, such as fashion products, thereby limiting the scope of study on in-depth visual reasoning in rich image and language contexts. To address this issue, we collect the Compose Image Retrieval on Real-life images (CIRR) dataset, which consists of over 36,000 pairs of crowd-sourced, open-domain images with human-generated modifying text. To extend current methods to the open-domain, we propose CIRPLANT, a transformer based model that leverages rich pre-trained vision-and-language (V&L) knowledge for modifying visual features conditioned on natural language. Retrieval is then done by nearest neighbor lookup on the modified features. We demonstrate that with a relatively simple architecture, CIRPLANT outperforms existing methods on open-domain images, while match

Stats

• citations188
• S2 citations—
• github stars0
• HF likes0
• heat score17.07
• arxiv keyliu2021image

Related papers

• Instance-level Composed Image Retrieval (2025)0.00
• Scaling Prompt Instructed Zero Shot Composed Image Retrieval With Image-only Data (2025)0.00
• Context-cir: Learning From Concepts In Text For Composed Image Retrieval (2025)4.67
• Good4cir: Generating Detailed Synthetic Captions For Composed Image Retrieval (2025)0.00
• Cir-cot: Towards Interpretable Composed Image Retrieval Via End-to-end Chain-of-thought Reasoning (2025)0.00
• Leveraging Large Vision-language Model As User Intent-aware Encoder For Composed Image Retrieval (2024)3.58
• Zero-shot Composed Text-image Retrieval (2023)0.00
• Composed Image Retrieval Using Contrastive Learning And Task-oriented Clip-based Features (2023)16.84