LCD: Learned Cross-domain Descriptors For 2D-3D Matching

Abstract

In this work, we present a novel method to learn a local cross-domain descriptor for 2D image and 3D point cloud matching. Our proposed method is a dual auto-encoder neural network that maps 2D and 3D input into a shared latent space representation. We show that such local cross-domain descriptors in the shared embedding are more discriminative than those obtained from individual training in 2D and 3D domains. To facilitate the training process, we built a new dataset by collecting \(\approx 1.4\) millions of 2D-3D correspondences with various lighting conditions and settings from publicly available RGB-D scenes. Our descriptor is evaluated in three main experiments: 2D-3D matching, cross-domain retrieval, and sparse-to-dense depth estimation. Experimental results confirm the robustness of our approach as well as its competitive performance not only in solving cross-domain tasks but also in being able to generalize to solve sole 2D and 3D tasks. Our dataset and code are released public

Stats

• citations67
• S2 citations—
• github stars0
• HF likes0
• heat score13.74
• arxiv keypham2019lcd

Related papers

• Deeppoint3d: Learning Discriminative Local Descriptors Using Deep Metric Learning On 3D Point Clouds (2019)9.59
• Learning Local Descriptors By Optimizing The Keypoint-correspondence Criterion: Applications To Face Matching, Learning From Unlabeled Videos And 3d-shape Retrieval (2016)11.75
• Crossloc3d: Aerial-ground Cross-source 3D Place Recognition (2023)9.23
• Cross-domain Visual Matching Via Generalized Similarity Measure And Feature Learning (2016)15.54
• DH3D: Deep Hierarchical 3D Descriptors For Robust Large-scale 6dof Relocalization (2020)14.76
• End-to-end Learning Of Keypoint Detector And Descriptor For Pose Invariant 3D Matching (2018)12.40
• Pn-net: Conjoined Triple Deep Network For Learning Local Image Descriptors (2016)0.00
• Hierarchical Metric Learning And Matching For 2D And 3D Geometric Correspondences (2018)11.39