Parts2words: Learning Joint Embedding Of Point Clouds And Texts By Bidirectional Matching Between Parts And Words

Abstract

Shape-Text matching is an important task of high-level shape understanding. Current methods mainly represent a 3D shape as multiple 2D rendered views, which obviously can not be understood well due to the structural ambiguity caused by self-occlusion in the limited number of views. To resolve this issue, we directly represent 3D shapes as point clouds, and propose to learn joint embedding of point clouds and texts by bidirectional matching between parts from shapes and words from texts. Specifically, we first segment the point clouds into parts, and then leverage optimal transport method to match parts and words in an optimized feature space, where each part is represented by aggregating features of all points within it and each word is abstracted by its contextual information. We optimize the feature space in order to enlarge the similarities between the paired training samples, while simultaneously maximizing the margin between the unpaired ones. Experiments demonstrate that our meth

Stats

• citations9
• S2 citations—
• github stars16
• HF likes0
• heat score9.96
• arxiv keytang2021parts2words

Related papers

• Coe: Deep Coupled Embedding For Non-rigid Point Cloud Correspondences (2024)0.00
• Y^2seq2seq: Cross-modal Representation Learning For 3D Shape And Text By Joint Reconstruction And Prediction Of View And Word Sequences (2018)12.02
• Joint Learning Of 3D Shape Retrieval And Deformation (2021)11.08
• Tricolo: Trimodal Contrastive Loss For Text To Shape Retrieval (2022)8.09
• Pointcloud-text Matching: Benchmark Datasets And A Baseline (2024)2.26
• Improving Text-based Person Search Via Part-level Cross-modal Correspondence (2024)0.00
• COM3D: Leveraging Cross-view Correspondence And Cross-modal Mining For 3D Retrieval (2024)3.58
• Deep Multimodal Image-text Embeddings For Automatic Cross-media Retrieval (2020)0.00