Fine-tuned But Zero-shot 3D Shape Sketch View Similarity And Retrieval

Abstract

Recently, encoders like ViT (vision transformer) and ResNet have been trained on vast datasets and utilized as perceptual metrics for comparing sketches and images, as well as multi-domain encoders in a zero-shot setting. However, there has been limited effort to quantify the granularity of these encoders. Our work addresses this gap by focusing on multi-modal 2D projections of individual 3D instances. This task holds crucial implications for retrieval and sketch-based modeling. We show that in a zero-shot setting, the more abstract the sketch, the higher the likelihood of incorrect image matches. Even within the same sketch domain, sketches of the same object drawn in different styles, for example by distinct individuals, might not be accurately matched. One of the key findings of our research is that meticulous fine-tuning on one class of 3D shapes can lead to improved performance on other shape classes, reaching or surpassing the accuracy of supervised methods. We compare and discus

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyberardi2023fine

Related papers

• Zero In On Shape: A Generic 2D-3D Instance Similarity Metric Learned From Synthetic Data (2021)5.84
• Structure-aware 3D VR Sketch To 3D Shape Retrieval (2022)8.94
• Towards 3D Vr-sketch To 3D Shape Retrieval (2022)7.16
• CLIP For All Things Zero-shot Sketch-based Image Retrieval, Fine-grained Or Not (2023)15.54
• Diff-sbsr: Learning Multimodal Feature-enhanced Diffusion Models For Zero-shot Sketch-based 3D Shape Retrieval (2026)0.00
• Domain-smoothing Network For Zero-shot Sketch-based Image Retrieval (2021)13.92
• Sketch3t: Test-time Training For Zero-shot SBIR (2022)13.23
• Freeview Sketching: View-aware Fine-grained Sketch-based Image Retrieval (2024)6.34