Sketchtriplet: Self-supervised Scenarized Sketch-text-image Triplet Generation

Abstract

The scarcity of free-hand sketch presents a challenging problem. Despite the emergence of some large-scale sketch datasets, these datasets primarily consist of sketches at the single-object level. There continues to be a lack of large-scale paired datasets for scene sketches. In this paper, we propose a self-supervised method for scene sketch generation that does not rely on any existing scene sketch, enabling the transformation of single-object sketches into scene sketches. To accomplish this, we introduce a method for vector sketch captioning and sketch semantic expansion. Additionally, we design a sketch generation network that incorporates a fusion of multi-modal perceptual constraints, suitable for application in zero-shot image-to-sketch downstream task, demonstrating state-of-the-art performance through experimental validation. Finally, leveraging our proposed sketch-to-sketch generation method, we contribute a large-scale dataset centered around scene sketches, comprising highl

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keywu2024sketchtriplet

Related papers

• Deep Self-supervised Representation Learning For Free-hand Sketch (2020)12.25
• FS-COCO: Towards Understanding Of Freehand Sketches Of Common Objects In Context (2022)11.93
• Scenetrilogy: On Human Scene-sketch And Its Complementarity With Photo And Text (2022)10.35
• Instance-level Sketch-based Retrieval By Deep Triplet Classification Siamese Network (2018)0.00
• Back To The Drawing Board: Rethinking Scene-level Sketch-based Image Retrieval (2025)0.00
• Triplet-aware Scene Graph Embeddings (2019)7.81
• Generalisation And Sharing In Triplet Convnets For Sketch Based Visual Search (2016)13.11
• Sketch3t: Test-time Training For Zero-shot SBIR (2022)13.23