Contrastive Video-language Learning With Fine-grained Frame Sampling

Abstract

Despite recent progress in video and language representation learning, the weak or sparse correspondence between the two modalities remains a bottleneck in the area. Most video-language models are trained via pair-level loss to predict whether a pair of video and text is aligned. However, even in paired video-text segments, only a subset of the frames are semantically relevant to the corresponding text, with the remainder representing noise; where the ratio of noisy frames is higher for longer videos. We propose FineCo (Fine-grained Contrastive Loss for Frame Sampling), an approach to better learn video and language representations with a fine-grained contrastive objective operating on video frames. It helps distil a video by selecting the frames that are semantically equivalent to the text, improving cross-modal correspondence. Building on the well established VideoCLIP model as a starting point, FineCo achieves state-of-the-art performance on YouCookII, a text-video retrieval benchma

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keywang2022contrastive

Related papers

• Fine-tuned CLIP Models Are Efficient Video Learners (2022)21.57
• X-CLIP: End-to-end Multi-grained Contrastive Learning For Video-text Retrieval (2022)18.12
• FG-CLIP: Fine-grained Visual And Textual Alignment (2025)5.75
• Towards Fast Adaptation Of Pretrained Contrastive Models For Multi-channel Video-language Retrieval (2022)7.50
• Crossclr: Cross-modal Contrastive Learning For Multi-modal Video Representations (2021)15.59
• TCLR: Temporal Contrastive Learning For Video Representation (2021)15.78
• Expectation-maximization Contrastive Learning For Compact Video-and-language Representations (2022)2.26
• Clip2video: Mastering Video-text Retrieval Via Image CLIP (2021)0.00