Video-text Pre-training With Learned Regions

Abstract

Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs via aligning the semantics between visual and textual information. State-of-the-art approaches extract visual features from raw pixels in an end-to-end fashion. However, these methods operate at frame-level directly and thus overlook the spatio-temporal structure of objects in video, which yet has a strong synergy with nouns in textual descriptions. In this work, we propose a simple yet effective module for video-text representation learning, namely RegionLearner, which can take into account the structure of objects during pre-training on large-scale video-text pairs. Given a video, our module (1) first quantizes visual features into semantic clusters, then (2) generates learnable masks and uses them to aggregate the features belonging to the same semantic region, and finally (3) models the interactions between different aggregated regions. In contrast to using off-the-shelf object d

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyan2021video

Related papers

• Revitalize Region Feature For Democratizing Video-language Pre-training Of Retrieval (2022)2.72
• Temporal Perceiving Video-language Pre-training (2023)0.00
• Locvtp: Video-text Pre-training For Temporal Localization (2022)11.39
• Text-video Retrieval With Global-local Semantic Consistent Learning (2024)8.75
• MILES: Visual BERT Pre-training With Injected Language Semantics For Video-text Retrieval (2022)10.61
• Clip2video: Mastering Video-text Retrieval Via Image CLIP (2021)0.00
• Masked Contrastive Pre-training For Efficient Video-text Retrieval (2022)5.84
• Rap: Redundancy-aware Video-language Pre-training For Text-video Retrieval (2022)7.05