Cap4video: What Can Auxiliary Captions Do For Text-video Retrieval?

Abstract

Most existing text-video retrieval methods focus on cross-modal matching between the visual content of videos and textual query sentences. However, in real-world scenarios, online videos are often accompanied by relevant text information such as titles, tags, and even subtitles, which can be utilized to match textual queries. This insight has motivated us to propose a novel approach to text-video retrieval, where we directly generate associated captions from videos using zero-shot video captioning with knowledge from web-scale pre-trained models (e.g., CLIP and GPT-2). Given the generated captions, a natural question arises: what benefits do they bring to text-video retrieval? To answer this, we introduce Cap4Video, a new framework that leverages captions in three ways: i) Input data: video-caption pairs can augment the training data. ii) Intermediate feature interaction: we perform cross-modal feature interaction between the video and caption to produce enhanced video representations.

Stats

• citations112
• S2 citations—
• github stars258
• HF likes0
• heat score20.22
• arxiv keywu2022cap4video

Related papers

• Narrating The Video: Boosting Text-video Retrieval Via Comprehensive Utilization Of Frame-level Captions (2025)6.77
• Clip4clip: An Empirical Study Of CLIP For End To End Video Clip Retrieval (2021)6.02
• Prompt Switch: Efficient CLIP Adaptation For Text-video Retrieval (2023)11.93
• Learning Audio-video Modalities From Image Captions (2022)12.54
• CLIPS: An Enhanced CLIP Framework For Learning With Synthetic Captions (2024)0.00
• CLIP2TV: Align, Match And Distill For Video-text Retrieval (2021)0.00
• Clip2video: Mastering Video-text Retrieval Via Image CLIP (2021)0.00
• Tencent Text-video Retrieval: Hierarchical Cross-modal Interactions With Multi-level Representations (2022)7.81