Clover: Towards A Unified Video-language Alignment And Fusion Model

Abstract

Building a universal Video-Language model for solving various video understanding tasks (*e.g.*, text-video retrieval, video question answering) is an open challenge to the machine learning field. Towards this goal, most recent works build the model by stacking uni-modal and cross-modal feature encoders and train it with pair-wise contrastive pre-text tasks. Though offering attractive generality, the resulted models have to compromise between efficiency and performance. They mostly adopt different architectures to deal with different downstream tasks. We find this is because the pair-wise training cannot well *align* and *fuse* features from different modalities. We then introduce \textbf\{Clover\}\textemdash a Correlated Video-Language pre-training method\textemdash towards a universal Video-Language model for solving multiple video understanding tasks with neither performance nor efficiency compromise. It improves cross-modal feature alignment and fusion via a novel tri-modal alignme

Stats

• citations29
• S2 citations—
• github stars40
• HF likes0
• heat score14.30
• arxiv keyhuang2022clover

Related papers

• Towards Fast Adaptation Of Pretrained Contrastive Models For Multi-channel Video-language Retrieval (2022)7.50
• Unicoder-vl: A Universal Encoder For Vision And Language By Cross-modal Pre-training (2019)20.24
• Clip-vip: Adapting Pre-trained Image-text Model To Video-language Representation Alignment (2022)5.42
• Everything At Once -- Multi-modal Fusion Transformer For Video Retrieval (2021)15.78
• Come-vl: Scaling Complementary Multi-encoder Vision-language Learning (2026)0.00
• A Multi-level Alignment Training Scheme For Video-and-language Grounding (2022)3.58
• Cross-view Language Modeling: Towards Unified Cross-lingual Cross-modal Pre-training (2022)8.09
• Unifying Latent And Lexicon Representations For Effective Video-text Retrieval (2024)0.00