Cross-modal Hierarchical Modelling For Fine-grained Sketch Based Image Retrieval

Abstract

Sketch as an image search query is an ideal alternative to text in capturing the fine-grained visual details. Prior successes on fine-grained sketch-based image retrieval (FG-SBIR) have demonstrated the importance of tackling the unique traits of sketches as opposed to photos, e.g., temporal vs. static, strokes vs. pixels, and abstract vs. pixel-perfect. In this paper, we study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail -- a person typically sketches up to various extents of detail to depict an object. This hierarchical structure is often visually distinct. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels. In particular, features from a sketch and a photo are enriched using cross-modal co-attention, coupled with hierarchical node fusion at every level to form a better emb

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keysain2020cross

Related papers

• Sketch Less For More: On-the-fly Fine-grained Sketch Based Image Retrieval (2020)15.28
• Cross-modal Subspace Learning For Fine-grained Sketch-based Image Retrieval (2017)13.34
• Sketch And Text Synergy: Fusing Structural Contours And Descriptive Attributes For Fine-grained Image Retrieval (2026)0.00
• Freeview Sketching: View-aware Fine-grained Sketch-based Image Retrieval (2024)6.34
• More Photos Are All You Need: Semi-supervised Learning For Fine-grained Sketch Based Image Retrieval (2021)13.23
• Deep Reinforced Attention Regression For Partial Sketch Based Image Retrieval (2021)5.24
• You'll Never Walk Alone: A Sketch And Text Duet For Fine-grained Image Retrieval (2024)9.41
• Back To The Drawing Board: Rethinking Scene-level Sketch-based Image Retrieval (2025)0.00