SAMURAI: Shape-aware Multimodal Retrieval For 3D Object Identification

Abstract

Retrieving 3D objects in complex indoor environments using only a masked 2D image and a natural language description presents significant challenges. The ROOMELSA challenge limits access to full 3D scene context, complicating reasoning about object appearance, geometry, and semantics. These challenges are intensified by distorted viewpoints, textureless masked regions, ambiguous language prompts, and noisy segmentation masks. To address this, we propose SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification. SAMURAI integrates CLIP-based semantic matching with shape-guided re-ranking derived from binary silhouettes of masked regions, alongside a robust majority voting strategy. A dedicated preprocessing pipeline enhances mask quality by extracting the largest connected component and removing background noise. Our hybrid retrieval framework leverages both language and shape cues, achieving competitive performance on the ROOMELSA private test set. These results highlight

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvo2025samurai

Related papers

• SHREC 2025: Retrieval Of Optimal Objects For Multi-modal Enhanced Language And Spatial Assistance (ROOMELSA) (2025)3.58
• SCA3D: Enhancing Cross-modal 3D Retrieval Via 3D Shape And Caption Paired Data Augmentation (2025)4.17
• COM3D: Leveraging Cross-view Correspondence And Cross-modal Mining For 3D Retrieval (2024)3.58
• OSCAR: Open-set CAD Retrieval From A Language Prompt And A Single Image (2026)0.00
• Mask2cad: 3D Shape Prediction By Learning To Segment And Retrieve (2020)12.87
• Describe, Adapt And Combine: Empowering CLIP Encoders For Open-set 3D Object Retrieval (2025)2.51
• Enhanced Cross-modal 3D Retrieval Via Tri-modal Reconstruction (2025)0.00
• Ri-mamba: Rotation-invariant Mamba For Robust Text-to-shape Retrieval (2026)0.00