Pic2word: Mapping Pictures To Words For Zero-shot Composed Image Retrieval

Abstract

In Composed Image Retrieval (CIR), a user combines a query image with text to describe their intended target. Existing methods rely on supervised learning of CIR models using labeled triplets consisting of the query image, text specification, and the target image. Labeling such triplets is expensive and hinders broad applicability of CIR. In this work, we propose to study an important task, Zero-Shot Composed Image Retrieval (ZS-CIR), whose goal is to build a CIR model without requiring labeled triplets for training. To this end, we propose a novel method, called Pic2Word, that requires only weakly labeled image-caption pairs and unlabeled image datasets to train. Unlike existing supervised CIR models, our model trained on weakly labeled or unlabeled datasets shows strong generalization across diverse ZS-CIR tasks, e.g., attribute editing, object composition, and domain conversion. Our approach outperforms several supervised CIR methods on the common CIR benchmark, CIRR and Fashion-IQ.

Stats

• citations121
• S2 citations—
• github stars196
• HF likes0
• heat score20.24
• arxiv keysaito2023pic2word

Related papers

• From Mapping To Composing: A Two-stage Framework For Zero-shot Composed Image Retrieval (2025)0.00
• Context-i2w: Mapping Images To Context-dependent Words For Accurate Zero-shot Composed Image Retrieval (2023)15.41
• Denoise-i2w: Mapping Images To Denoising Words For Accurate Zero-shot Composed Image Retrieval (2024)1.40
• Isearle: Improving Textual Inversion For Zero-shot Composed Image Retrieval (2024)12.09
• Zero-shot Composed Image Retrieval With Textual Inversion (2023)19.84
• Image2sentence Based Asymmetrical Zero-shot Composed Image Retrieval (2024)0.00
• Fine-grained Zero-shot Composed Image Retrieval With Complementary Visual-semantic Integration (2026)1.24
• Training-free Zero-shot Composed Image Retrieval Via Weighted Modality Fusion And Similarity (2024)5.84