A Sketch Is Worth A Thousand Words: Image Retrieval With Text And Sketch

Abstract

We address the problem of retrieving images with both a sketch and a text query. We present TASK-former (Text And SKetch transformer), an end-to-end trainable model for image retrieval using a text description and a sketch as input. We argue that both input modalities complement each other in a manner that cannot be achieved easily by either one alone. TASK-former follows the late-fusion dual-encoder approach, similar to CLIP, which allows efficient and scalable retrieval since the retrieval set can be indexed independently of the queries. We empirically demonstrate that using an input sketch (even a poorly drawn one) in addition to text considerably increases retrieval recall compared to traditional text-based image retrieval. To evaluate our approach, we collect 5,000 hand-drawn sketches for images in the test set of the COCO dataset. The collected sketches are available a https://janesjanes.github.io/tsbir/.

Stats

• citations23
• S2 citations—
• github stars0
• HF likes0
• heat score10.35
• arxiv keysangkloy2022a

Related papers

• You'll Never Walk Alone: A Sketch And Text Duet For Fine-grained Image Retrieval (2024)9.41
• Sketch And Text Synergy: Fusing Structural Contours And Descriptive Attributes For Fine-grained Image Retrieval (2026)0.00
• Composite Sketch+text Queries For Retrieving Objects With Elusive Names And Complex Interactions (2025)5.84
• Back To The Drawing Board: Rethinking Scene-level Sketch-based Image Retrieval (2025)0.00
• Sketching Without Worrying: Noise-tolerant Sketch-based Image Retrieval (2022)12.74
• Learning Cross-modal Deep Embeddings For Multi-object Image Retrieval Using Text And Sketch (2018)9.59
• FS-COCO: Towards Understanding Of Freehand Sketches Of Common Objects In Context (2022)11.93
• Sketch Less For More: On-the-fly Fine-grained Sketch Based Image Retrieval (2020)15.28