A Multi-level Alignment Training Scheme For Video-and-language Grounding

Abstract

To solve video-and-language grounding tasks, the key is for the network to understand the connection between the two modalities. For a pair of video and language description, their semantic relation is reflected by their encodings' similarity. A good multi-modality encoder should be able to well capture both inputs' semantics and encode them in the shared feature space where embedding distance gets properly translated into their semantic similarity. In this work, we focused on this semantic connection between video and language, and developed a multi-level alignment training scheme to directly shape the encoding process. Global and segment levels of video-language alignment pairs were designed, based on the information similarity ranging from high-level context to fine-grained semantics. The contrastive loss was used to contrast the encodings' similarities between the positive and negative alignment pairs, and to ensure the network is trained in such a way that similar information is e

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keyzhang2022a

Related papers

• Video-language Alignment Via Spatio-temporal Graph Transformer (2024)0.00
• Clover: Towards A Unified Video-language Alignment And Fusion Model (2022)14.30
• Tagging Before Alignment: Integrating Multi-modal Tags For Video-text Retrieval (2023)10.74
• Hanet: Hierarchical Alignment Networks For Video-text Retrieval (2021)0.00
• Modest-align: Data-efficient Alignment For Vision-language Models (2025)0.00
• Come-vl: Scaling Complementary Multi-encoder Vision-language Learning (2026)0.00
• Improving Video-text Retrieval By Multi-stream Corpus Alignment And Dual Softmax Loss (2021)0.00
• Towards Balanced Alignment: Modal-enhanced Semantic Modeling For Video Moment Retrieval (2023)14.33