Text-video Retrieval Via Variational Multi-modal Hypergraph Networks

Abstract

Text-video retrieval is a challenging task that aims to identify relevant videos given textual queries. Compared to conventional textual retrieval, the main obstacle for text-video retrieval is the semantic gap between the textual nature of queries and the visual richness of video content. Previous works primarily focus on aligning the query and the video by finely aggregating word-frame matching signals. Inspired by the human cognitive process of modularly judging the relevance between text and video, the judgment needs high-order matching signal due to the consecutive and complex nature of video contents. In this paper, we propose chunk-level text-video matching, where the query chunks are extracted to describe a specific retrieval unit, and the video chunks are segmented into distinct clips from videos. We formulate the chunk-level matching as n-ary correlations modeling between words of the query and frames of the video and introduce a multi-modal hypergraph for n-ary correlation m

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyli2024text

Related papers

• Fine-grained Video-text Retrieval With Hierarchical Graph Reasoning (2020)18.27
• Tencent Text-video Retrieval: Hierarchical Cross-modal Interactions With Multi-level Representations (2022)7.81
• Hanet: Hierarchical Alignment Networks For Video-text Retrieval (2021)0.00
• Text-adaptive Multiple Visual Prototype Matching For Video-text Retrieval (2022)4.52
• Delving Deeper: Hierarchical Visual Perception For Robust Video-text Retrieval (2026)1.24
• Dual Encoding For Video Retrieval By Text (2020)16.05
• Frame-wise Cross-modal Matching For Video Moment Retrieval (2020)13.17
• UATVR: Uncertainty-adaptive Text-video Retrieval (2023)15.46