Mask2cad: 3D Shape Prediction By Learning To Segment And Retrieve

Abstract

Object recognition has seen significant progress in the image domain, with focus primarily on 2D perception. We propose to leverage existing large-scale datasets of 3D models to understand the underlying 3D structure of objects seen in an image by constructing a CAD-based representation of the objects and their poses. We present Mask2CAD, which jointly detects objects in real-world images and for each detected object, optimizes for the most similar CAD model and its pose. We construct a joint embedding space between the detected regions of an image corresponding to an object and 3D CAD models, enabling retrieval of CAD models for an input RGB image. This produces a clean, lightweight representation of the objects in an image; this CAD-based representation ensures a valid, efficient shape representation for applications such as content creation or interactive scenarios, and makes a step towards understanding the transformation of real-world imagery to a synthetic domain. Experiments on

Stats

• citations51
• S2 citations—
• github stars0
• HF likes0
• heat score12.87
• arxiv keykuo2020mask2cad

Related papers

• Patch2cad: Patchwise Embedding Learning For In-the-wild Shape Retrieval From A Single Image (2021)10.85
• Fastcad: Real-time CAD Retrieval And Alignment From Scans And Videos (2024)6.34
• ROCA: Robust CAD Model Retrieval And Alignment From A Single Image (2021)12.61
• Weakly-supervised End-to-end CAD Retrieval To Scan Objects (2022)0.00
• Joint Embedding Of 3D Scan And CAD Objects (2019)11.08
• Self-supervised Graph Neural Network For Mechanical CAD Retrieval (2024)0.00
• OSCAR: Open-set CAD Retrieval From A Language Prompt And A Single Image (2026)0.00
• 3D Pose Estimation And 3D Model Retrieval For Objects In The Wild (2018)15.25