Scene Designer: A Unified Model For Scene Search And Synthesis From Sketch

Abstract

Scene Designer is a novel method for searching and generating images using free-hand sketches of scene compositions; i.e. drawings that describe both the appearance and relative positions of objects. Our core contribution is a single unified model to learn both a cross-modal search embedding for matching sketched compositions to images, and an object embedding for layout synthesis. We show that a graph neural network (GNN) followed by Transformer under our novel contrastive learning setting is required to allow learning correlations between object type, appearance and arrangement, driving a mask generation module that synthesises coherent scene layouts, whilst also delivering state of the art sketch based visual search of scenes.

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyribeiro2021scene

Related papers

• Back To The Drawing Board: Rethinking Scene-level Sketch-based Image Retrieval (2025)0.00
• Scenetrilogy: On Human Scene-sketch And Its Complementarity With Photo And Text (2022)10.35
• Sketchtriplet: Self-supervised Scenarized Sketch-text-image Triplet Generation (2024)4.52
• Enhancing Product Search Interfaces With Sketch-guided Diffusion And Language Agents (2025)0.00
• Compact Scene Graphs For Layout Composition And Patch Retrieval (2019)8.09
• Sgdraw: Scene Graph Drawing Interface Using Object-oriented Representation (2022)0.00
• SCENIR: Visual Semantic Clarity Through Unsupervised Scene Graph Retrieval (2025)0.00
• FS-COCO: Towards Understanding Of Freehand Sketches Of Common Objects In Context (2022)11.93