Rome: Role-aware Mixture-of-expert Transformer For Text-to-video Retrieval

Abstract

Seas of videos are uploaded daily with the popularity of social channels; thus, retrieving the most related video contents with user textual queries plays a more crucial role. Most methods consider only one joint embedding space between global visual and textual features without considering the local structures of each modality. Some other approaches consider multiple embedding spaces consisting of global and local features separately, ignoring rich inter-modality correlations. We propose a novel mixture-of-expert transformer RoME that disentangles the text and the video into three levels; the roles of spatial contexts, temporal contexts, and object contexts. We utilize a transformer-based attention mechanism to fully exploit visual and text embeddings at both global and local levels with mixture-of-experts for considering inter-modalities and structures' correlations. The results indicate that our method outperforms the state-of-the-art methods on the YouCook2 and MSR-VTT datasets,

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keysatar2022rome

Related papers

• Semantic Role Aware Correlation Transformer For Text To Video Retrieval (2022)6.34
• Multi-modal Transformer For Video Retrieval (2020)19.47
• MDMMT-2: Multidomain Multimodal Transformer For Video Retrieval, One More Step Towards Generalization (2022)0.00
• Everything At Once -- Multi-modal Fusion Transformer For Video Retrieval (2021)15.78
• CLIP2TV: Align, Match And Distill For Video-text Retrieval (2021)0.00
• MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion (2025)2.26
• Memory Enhanced Embedding Learning For Cross-modal Video-text Retrieval (2021)0.00
• MERLIN: Multimodal Embedding Refinement Via Llm-based Iterative Navigation For Text-video Retrieval-rerank Pipeline (2024)5.84