Y^2seq2seq: Cross-modal Representation Learning For 3D Shape And Text By Joint Reconstruction And Prediction Of View And Word Sequences

Abstract

A recent method employs 3D voxels to represent 3D shapes, but this limits the approach to low resolutions due to the computational cost caused by the cubic complexity of 3D voxels. Hence the method suffers from a lack of detailed geometry. To resolve this issue, we propose Y^2Seq2Seq, a view-based model, to learn cross-modal representations by joint reconstruction and prediction of view and word sequences. Specifically, the network architecture of Y^2Seq2Seq bridges the semantic meaning embedded in the two modalities by two coupled `Y' like sequence-to-sequence (Seq2Seq) structures. In addition, our novel hierarchical constraints further increase the discriminability of the cross-modal representations by employing more detailed discriminative information. Experimental results on cross-modal retrieval and 3D shape captioning show that Y^2Seq2Seq outperforms the state-of-the-art methods.

Stats

• citations39
• S2 citations—
• github stars0
• HF likes0
• heat score12.02
• arxiv keyhan2018y

Related papers

• Enhanced Cross-modal 3D Retrieval Via Tri-modal Reconstruction (2025)0.00
• SCA3D: Enhancing Cross-modal 3D Retrieval Via 3D Shape And Caption Paired Data Augmentation (2025)4.17
• Parts2words: Learning Joint Embedding Of Point Clouds And Texts By Bidirectional Matching Between Parts And Words (2021)9.96
• Learning A Hierarchical Latent-variable Model Of 3D Shapes (2017)12.87
• Poseembroider: Towards A 3D, Visual, Semantic-aware Human Pose Representation (2024)6.34
• Tricolo: Trimodal Contrastive Loss For Text To Shape Retrieval (2022)8.09
• COM3D: Leveraging Cross-view Correspondence And Cross-modal Mining For 3D Retrieval (2024)3.58
• Gram Regularization For Multi-view 3D Shape Retrieval (2020)0.00