Mask To Reconstruct: Cooperative Semantics Completion For Video-text Retrieval

Abstract

Recently, masked video modeling has been widely explored and significantly improved the model's understanding ability of visual regions at a local level. However, existing methods usually adopt random masking and follow the same reconstruction paradigm to complete the masked regions, which do not leverage the correlations between cross-modal content. In this paper, we present Mask for Semantics Completion (MASCOT) based on semantic-based masked modeling. Specifically, after applying attention-based video masking to generate high-informed and low-informed masks, we propose Informed Semantics Completion to recover masked semantics information. The recovery mechanism is achieved by aligning the masked content with the unmasked visual regions and corresponding textual context, which makes the model capture more text-related details at a patch level. Additionally, we shift the emphasis of reconstruction from irrelevant backgrounds to discriminative parts to ignore regions with low-informed

Stats

• citations4
• S2 citations—
• github stars0
• HF likes0
• heat score5.24
• arxiv keyfang2023mask

Related papers

• MILES: Visual BERT Pre-training With Injected Language Semantics For Video-text Retrieval (2022)10.61
• Masked Contrastive Pre-training For Efficient Video-text Retrieval (2022)5.84
• E-vilm: Efficient Video-language Model Via Masked Video Modeling With Semantic Vector-quantized Tokenizer (2023)0.00
• Towards Balanced Alignment: Modal-enhanced Semantic Modeling For Video Moment Retrieval (2023)14.33
• Boosting Video-text Retrieval With Explicit High-level Semantics (2022)7.50
• Masked Contrastive Reconstruction For Cross-modal Medical Image-report Retrieval (2023)0.00
• Text Is MASS: Modeling As Stochastic Embedding For Text-video Retrieval (2024)13.11
• Prota: Probabilistic Token Aggregation For Text-video Retrieval (2024)4.52