Video-text Retrieval By Supervised Sparse Multi-grained Learning

Abstract

While recent progress in video-text retrieval has been advanced by the exploration of better representation learning, in this paper, we present a novel multi-grained sparse learning framework, S3MA, to learn an aligned sparse space shared between the video and the text for video-text retrieval. The shared sparse space is initialized with a finite number of sparse concepts, each of which refers to a number of words. With the text data at hand, we learn and update the shared sparse space in a supervised manner using the proposed similarity and alignment losses. Moreover, to enable multi-grained alignment, we incorporate frame representations for better modeling the video modality and calculating fine-grained and coarse-grained similarities. Benefiting from the learned shared sparse space and multi-grained similarities, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of S3MA over existing methods. Our code is available at https://github.com/yim

Stats

• citations6
• S2 citations—
• github stars6
• HF likes0
• heat score8.03
• arxiv keywang2023video

Related papers

• Improving Video-text Retrieval By Multi-stream Corpus Alignment And Dual Softmax Loss (2021)0.00
• TC-MGC: Text-conditioned Multi-grained Contrastive Learning For Text-video Retrieval (2025)6.93
• Text-video Retrieval With Global-local Semantic Consistent Learning (2024)8.75
• X-CLIP: End-to-end Multi-grained Contrastive Learning For Video-text Retrieval (2022)18.12
• Text Is MASS: Modeling As Stochastic Embedding For Text-video Retrieval (2024)13.11
• T2VLAD: Global-local Sequence Alignment For Text-video Retrieval (2021)16.65
• Text-video Retrieval Via Variational Multi-modal Hypergraph Networks (2024)0.00
• Unifying Latent And Lexicon Representations For Effective Video-text Retrieval (2024)0.00