Zero In On Shape: A Generic 2D-3D Instance Similarity Metric Learned From Synthetic Data

Abstract

We present a network architecture which compares RGB images and untextured 3D models by the similarity of the represented shape. Our system is optimised for zero-shot retrieval, meaning it can recognise shapes never shown in training. We use a view-based shape descriptor and a siamese network to learn object geometry from pairs of 3D models and 2D images. Due to scarcity of datasets with exact photograph-mesh correspondences, we train our network with only synthetic data. Our experiments investigate the effect of different qualities and quantities of training data on retrieval accuracy and present insights from bridging the domain gap. We show that increasing the variety of synthetic data improves retrieval accuracy and that our system's performance in zero-shot mode can match that of the instance-aware mode, as far as narrowing down the search to the top 10% of objects.

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyjanik2021zero

Related papers

• Fine-tuned But Zero-shot 3D Shape Sketch View Similarity And Retrieval (2023)5.24
• Hard Example Generation By Texture Synthesis For Cross-domain Shape Similarity Learning (2020)0.95
• Deep Shape Matching (2017)12.61
• Extending Deepsdf For Automatic 3D Shape Retrieval And Similarity Transform Estimation (2020)0.00
• Risa-net: Rotation-invariant Structure-aware Network For Fine-grained 3D Shape Retrieval (2020)5.48
• Unsupervised Detection Of Distinctive Regions On 3D Shapes (2019)7.50
• Generalizing Single-view 3D Shape Retrieval To Occlusions And Unseen Objects (2023)5.24
• Optimizing Multi-modal Models For Image-based Shape Retrieval: The Role Of Pre-alignment And Hard Contrastive Learning (2026)0.00