MDMMT-2: Multidomain Multimodal Transformer For Video Retrieval, One More Step Towards Generalization

Abstract

In this work we present a new State-of-The-Art on the text-to-video retrieval task on MSR-VTT, LSMDC, MSVD, YouCook2 and TGIF obtained by a single model. Three different data sources are combined: weakly-supervised videos, crowd-labeled text-image pairs and text-video pairs. A careful analysis of available pre-trained networks helps to choose the best prior-knowledge ones. We introduce three-stage training procedure that provides high transfer knowledge efficiency and allows to use noisy datasets during training without prior knowledge degradation. Additionally, double positional encoding is used for better fusion of different modalities and a simple method for non-square inputs processing is suggested.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykunitsyn2022mdmmt

Related papers

• Multi-modal Transformer For Video Retrieval (2020)19.47
• Everything At Once -- Multi-modal Fusion Transformer For Video Retrieval (2021)15.78
• Mv-adapter: Multimodal Video Transfer Learning For Video Text Retrieval (2023)9.76
• CLIP2TV: Align, Match And Distill For Video-text Retrieval (2021)0.00
• Rome: Role-aware Mixture-of-expert Transformer For Text-to-video Retrieval (2022)0.00
• TEACHTEXT: Crossmodal Generalized Distillation For Text-video Retrieval (2021)15.43
• MMMORRF: Multimodal Multilingual Modularized Reciprocal Rank Fusion (2025)2.26
• Recurrence Meets Transformers For Universal Multimodal Retrieval (2025)2.41