Temporal Context Aggregation For Video Retrieval With Contrastive Learning

Abstract

The current research focus on Content-Based Video Retrieval requires higher-level video representation describing the long-range semantic dependencies of relevant incidents, events, etc. However, existing methods commonly process the frames of a video as individual images or short clips, making the modeling of long-range semantic dependencies difficult. In this paper, we propose TCA (Temporal Context Aggregation for Video Retrieval), a video representation learning framework that incorporates long-range temporal information between frame-level features using the self-attention mechanism. To train it on video retrieval datasets, we propose a supervised contrastive learning method that performs automatic hard negative mining and utilizes the memory bank mechanism to increase the capacity of negative samples. Extensive experiments are conducted on multiple video retrieval tasks, such as CC_WEB_VIDEO, FIVR-200K, and EVVE. The proposed method shows a significant performance advantage (~17%

Stats

• citations57
• S2 citations—
• github stars0
• HF likes0
• heat score13.23
• arxiv keyshao2020temporal

Related papers

• TCLR: Temporal Contrastive Learning For Video Representation (2021)15.78
• TC-MGC: Text-conditioned Multi-grained Contrastive Learning For Text-video Retrieval (2025)6.93
• Prota: Probabilistic Token Aggregation For Text-video Retrieval (2024)4.52
• Graph Based Temporal Aggregation For Video Retrieval (2020)0.00
• Relevance-based Margin For Contrastively-trained Video Retrieval Models (2022)7.74
• X-CLIP: End-to-end Multi-grained Contrastive Learning For Video-text Retrieval (2022)18.12
• Sequential Contrastive Audio-visual Learning (2024)5.84
• Towards Efficient And Robust Moment Retrieval System: A Unified Framework For Multi-granularity Models And Temporal Reranking (2025)2.26