Crossover: 3D Scene Cross-modal Alignment

Abstract

Multi-modal 3D object understanding has gained significant attention, yet current approaches often assume complete data availability and rigid alignment across all modalities. We present CrossOver, a novel framework for cross-modal 3D scene understanding via flexible, scene-level modality alignment. Unlike traditional methods that require aligned modality data for every object instance, CrossOver learns a unified, modality-agnostic embedding space for scenes by aligning modalities -- RGB images, point clouds, CAD models, floorplans, and text descriptions -- with relaxed constraints and without explicit object semantics. Leveraging dimensionality-specific encoders, a multi-stage training pipeline, and emergent cross-modal behaviors, CrossOver supports robust scene retrieval and object localization, even with missing modalities. Evaluations on ScanNet and 3RScan datasets show its superior performance across diverse metrics, highlighting the adaptability for real-world applications in 3D

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keysarkar2025crossover

Related papers

• Enhanced Cross-modal 3D Retrieval Via Tri-modal Reconstruction (2025)0.00
• COM3D: Leveraging Cross-view Correspondence And Cross-modal Mining For 3D Retrieval (2024)3.58
• Learning Aligned Cross-modal Representations From Weakly Aligned Data (2016)14.97
• Cross-modal Scene Networks (2016)14.35
• Crossloc3d: Aerial-ground Cross-source 3D Place Recognition (2023)9.23
• Discriminative Semantic Transitive Consistency For Cross-modal Learning (2021)0.00
• Cross-modal Center Loss (2020)0.00
• LCD: Learned Cross-domain Descriptors For 2D-3D Matching (2019)13.74