Generating Holistic 3D Scene Abstractions For Text-based Image Retrieval

Abstract

Spatial relationships between objects provide important information for text-based image retrieval. As users are more likely to describe a scene from a real world perspective, using 3D spatial relationships rather than 2D relationships that assume a particular viewing direction, one of the main challenges is to infer the 3D structure that bridges images with users' text descriptions. However, direct inference of 3D structure from images requires learning from large scale annotated data. Since interactions between objects can be reduced to a limited set of atomic spatial relations in 3D, we study the possibility of inferring 3D structure from a text description rather than an image, applying physical relation models to synthesize holistic 3D abstract object layouts satisfying the spatial constraints present in a textual description. We present a generic framework for retrieving images from a textual description of a scene by matching images with these generated abstract object layouts.

Stats

• citations15
• S2 citations—
• github stars0
• HF likes0
• heat score9.03
• arxiv keyli2016generating

Related papers

• Hyperbolic Hierarchical Alignment Reasoning Network For Text-3d Retrieval (2025)1.81
• Open-world 3D Scene Graph Generation For Retrieval-augmented Reasoning (2025)0.00
• COM3D: Leveraging Cross-view Correspondence And Cross-modal Mining For 3D Retrieval (2024)3.58
• Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020)17.18
• Purely Geometric Scene Association And Retrieval - A Case For Macro Scale 3D Geometry (2018)0.00
• SCA3D: Enhancing Cross-modal 3D Retrieval Via 3D Shape And Caption Paired Data Augmentation (2025)4.17
• Enhanced Cross-modal 3D Retrieval Via Tri-modal Reconstruction (2025)0.00
• Textanimar: Text-based 3D Animal Fine-grained Retrieval (2023)7.50