Fitclip: Refining Large-scale Pretrained Image-text Models For Zero-shot Video Understanding Tasks

Abstract

Large-scale pretrained image-text models have shown incredible zero-shot performance in a handful of tasks, including video ones such as action recognition and text-to-video retrieval. However, these models have not been adapted to video, mainly because they do not account for the time dimension but also because video frames are different from the typical images (e.g., containing motion blur, and less sharpness). In this paper, we present a fine-tuning strategy to refine these large-scale pretrained image-text models for zero-shot video understanding tasks. We show that by carefully adapting these models we obtain considerable improvements on two zero-shot Action Recognition tasks and three zero-shot Text-to-video Retrieval tasks. The code is available at https://github.com/bryant1410/fitclip

Stats

• citations0
• S2 citations—
• github stars8
• HF likes0
• heat score1.91
• arxiv keycastro2022fitclip

Code

• bryant1410/fitclip—

Related papers

• Videoclip: Contrastive Pre-training For Zero-shot Video-text Understanding (2021)28.04
• Fine-tuned CLIP Models Are Efficient Video Learners (2022)21.57
• Clip2video: Mastering Video-text Retrieval Via Image CLIP (2021)0.00
• Mobileclip: Fast Image-text Models Through Multi-modal Reinforced Training (2023)18.12
• Prompt Switch: Efficient CLIP Adaptation For Text-video Retrieval (2023)11.93
• Clip-vip: Adapting Pre-trained Image-text Model To Video-language Representation Alignment (2022)5.42
• Mobileviclip: An Efficient Video-text Model For Mobile Devices (2025)2.76
• Revisiting Temporal Modeling For Clip-based Image-to-video Knowledge Transferring (2023)17.40