Tri-modal Motion Retrieval By Learning A Joint Embedding Space

Abstract

Information retrieval is an ever-evolving and crucial research domain. The substantial demand for high-quality human motion data especially in online acquirement has led to a surge in human motion research works. Prior works have mainly concentrated on dual-modality learning, such as text and motion tasks, but three-modality learning has been rarely explored. Intuitively, an extra introduced modality can enrich a model's application scenario, and more importantly, an adequate choice of the extra modality can also act as an intermediary and enhance the alignment between the other two disparate modalities. In this work, we introduce LAVIMO (LAnguage-VIdeo-MOtion alignment), a novel framework for three-modality learning integrating human-centric videos as an additional modality, thereby effectively bridging the gap between text and motion. Moreover, our approach leverages a specially designed attention mechanism to foster enhanced alignment and synergistic effects among text, video, and m

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keyyin2024tri

Related papers

• Text-to-motion Retrieval: Towards Joint Understanding Of Human Motion Data And Natural Language (2023)11.94
• Joint Fusion And Encoding: Advancing Multimodal Retrieval From The Ground Up (2025)0.00
• Dual Encoding For Video Retrieval By Text (2020)16.05
• Verve: Versatile Retrieval For Videos Via Unified Embeddings (2026)0.00
• Unified Interactive Multimodal Moment Retrieval Via Cascaded Embedding-reranking And Temporal-aware Score Fusion (2025)0.00
• Multi-modal Transformer For Video Retrieval (2020)19.47
• Clamr: Contextualized Late-interaction For Multimodal Content Retrieval (2025)0.00
• Enhanced Cross-modal 3D Retrieval Via Tri-modal Reconstruction (2025)0.00