Semantic Role Aware Correlation Transformer For Text To Video Retrieval

Abstract

With the emergence of social media, voluminous video clips are uploaded every day, and retrieving the most relevant visual content with a language query becomes critical. Most approaches aim to learn a joint embedding space for plain textual and visual contents without adequately exploiting their intra-modality structures and inter-modality correlations. This paper proposes a novel transformer that explicitly disentangles the text and video into semantic roles of objects, spatial contexts and temporal contexts with an attention scheme to learn the intra- and inter-role correlations among the three roles to discover discriminative features for matching at different levels. The preliminary results on popular YouCook2 indicate that our approach surpasses a current state-of-the-art method, with a high margin in all metrics. It also overpasses two SOTA methods in terms of two metrics.

Stats

• citations6
• S2 citations—
• github stars0
• HF likes0
• heat score6.34
• arxiv keysatar2022semantic

Related papers

• Rome: Role-aware Mixture-of-expert Transformer For Text-to-video Retrieval (2022)0.00
• Multi-modal Transformer For Video Retrieval (2020)19.47
• MDMMT-2: Multidomain Multimodal Transformer For Video Retrieval, One More Step Towards Generalization (2022)0.00
• Contra: (con)text (tra)nsformer For Cross-modal Video Retrieval (2022)2.26
• CLIP2TV: Align, Match And Distill For Video-text Retrieval (2021)0.00
• Object-aware Video-language Pre-training For Retrieval (2021)16.14
• Exploiting Semantic Role Contextualized Video Features For Multi-instance Text-video Retrieval EPIC-KITCHENS-100 Multi-instance Retrieval Challenge 2022 (2022)0.00
• Hit: Hierarchical Transformer With Momentum Contrast For Video-text Retrieval (2021)15.98