CLIP For All Things Zero-shot Sketch-based Image Retrieval, Fine-grained Or Not
2023 · Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, et al.
Abstract
In this paper, we leverage CLIP for zero-shot sketch based image retrieval (ZS-SBIR). We are largely inspired by recent advances on foundation models and the unparalleled generalisation ability they seem to offer, but for the first time tailor it to benefit the sketch community. We put forward novel designs on how best to achieve this synergy, for both the category setting and the fine-grained setting ("all"). At the very core of our solution is a prompt learning setup. First we show just via factoring in sketch-specific prompts, we already have a category-level ZS-SBIR system that overshoots all prior arts, by a large margin (24.8%) - a great testimony on studying the CLIP and ZS-SBIR synergy. Moving onto the fine-grained setup is however trickier, and requires a deeper dive into this synergy. For that, we come up with two specific designs to tackle the fine-grained matching nature of the problem: (i) an additional regularisation loss to ensure the relative separation between sketches
Authors
(none)
Tags
Stats
Related papers
- Doodle To Search: Practical Zero-shot Sketch-based Image Retrieval (2019)16.75
- An Efficient Framework For Zero-shot Sketch-based Image Retrieval (2021)13.65
- Zero-shot Everything Sketch-based Image Retrieval, And In Explainable Style (2023)16.67
- Elevating All Zero-shot Sketch-based Image Retrieval Through Multimodal Prompt Learning (2024)6.34
- Adapt And Align To Improve Zero-shot Sketch-based Image Retrieval (2023)0.00
- A Zero-shot Framework For Sketch-based Image Retrieval (2018)16.49
- Relation-aware Meta-learning For Zero-shot Sketch-based Image Retrieval (2024)0.00
- Crossatnet - A Novel Cross-attention Based Framework For Sketch-based Image Retrieval (2021)11.29