Triplet-aware Scene Graph Embeddings

Abstract

Scene graphs have become an important form of structured knowledge for tasks such as for image generation, visual relation detection, visual question answering, and image retrieval. While visualizing and interpreting word embeddings is well understood, scene graph embeddings have not been fully explored. In this work, we train scene graph embeddings in a layout generation task with different forms of supervision, specifically introducing triplet super-vision and data augmentation. We see a significant performance increase in both metrics that measure the goodness of layout prediction, mean intersection-over-union (mIoU)(52.3% vs. 49.2%) and relation score (61.7% vs. 54.1%),after the addition of triplet supervision and data augmentation. To understand how these different methods affect the scene graph representation, we apply several new visualization and evaluation methods to explore the evolution of the scene graph embedding. We find that triplet supervision significantly improves the

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keyschroeder2019triplet

Related papers

• Scene Graph Embeddings Using Relative Similarity Supervision (2021)7.50
• Sketchtriplet: Self-supervised Scenarized Sketch-text-image Triplet Generation (2024)4.52
• Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020)17.18
• Compact Scene Graphs For Layout Composition And Patch Retrieval (2019)8.09
• Scenetrilogy: On Human Scene-sketch And Its Complementarity With Photo And Text (2022)10.35
• SCENIR: Visual Semantic Clarity Through Unsupervised Scene Graph Retrieval (2025)0.00
• Open-world 3D Scene Graph Generation For Retrieval-augmented Reasoning (2025)0.00
• Generalisation And Sharing In Triplet Convnets For Sketch Based Visual Search (2016)13.11