Sketch Less For More: On-the-fly Fine-grained Sketch Based Image Retrieval

Abstract

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. Its widespread applicability is however hindered by the fact that drawing a sketch takes time, and most people struggle to draw a complete and faithful sketch. In this paper, we reformulate the conventional FG-SBIR framework to tackle these challenges, with the ultimate goal of retrieving the target photo with the least number of strokes possible. We further propose an on-the-fly design that starts retrieving as soon as the user starts drawing. To accomplish this, we devise a reinforcement learning-based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode. Additionally, we introduce a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval. We achieve superior early

Stats

• citations108
• S2 citations—
• github stars0
• HF likes0
• heat score15.28
• arxiv keybhunia2020sketch

Related papers

• Active Learning For Fine-grained Sketch-based Image Retrieval (2023)0.00
• Deep Reinforced Attention Regression For Partial Sketch Based Image Retrieval (2021)5.24
• Freeview Sketching: View-aware Fine-grained Sketch-based Image Retrieval (2024)6.34
• More Photos Are All You Need: Semi-supervised Learning For Fine-grained Sketch Based Image Retrieval (2021)13.23
• Doodle To Search: Practical Zero-shot Sketch-based Image Retrieval (2019)16.75
• Adaptive Fine-grained Sketch-based Image Retrieval (2022)9.76
• Sketching Without Worrying: Noise-tolerant Sketch-based Image Retrieval (2022)12.74
• Back To The Drawing Board: Rethinking Scene-level Sketch-based Image Retrieval (2025)0.00