Vista: Vision And Scene Text Aggregation For Cross-modal Retrieval

Abstract

Visual appearance is considered to be the most important cue to understand images for cross-modal retrieval, while sometimes the scene text appearing in images can provide valuable information to understand the visual semantics. Most of existing cross-modal retrieval approaches ignore the usage of scene text information and directly adding this information may lead to performance degradation in scene text free scenarios. To address this issue, we propose a full transformer architecture to unify these cross-modal retrieval scenarios in a single \(\textbf\{Vi\}\)sion and \(\textbf\{S\}\)cene \(\textbf\{T\}\)ext \(\textbf\{A\}\)ggregation framework (ViSTA). Specifically, ViSTA utilizes transformer blocks to directly encode image patches and fuse scene text embedding to learn an aggregated visual representation for cross-modal retrieval. To tackle the modality missing problem of scene text, we propose a novel fusion token based transformer aggregation approach to exchange the necessary sce

Stats

• citations80
• S2 citations—
• github stars0
• HF likes0
• heat score14.31
• arxiv keycheng2022vista

Related papers

• VISTA: Visualized Text Embedding For Universal Multi-modal Retrieval (2024)16.73
• Beyond Visual Semantics: Exploring The Role Of Scene Text In Image Understanding (2019)9.59
• Stacmr: Scene-text Aware Cross-modal Retrieval (2020)10.48
• Scene Graph Based Fusion Network For Image-text Retrieval (2023)4.52
• Vldeformer: Vision-language Decomposed Transformer For Fast Cross-modal Retrieval (2021)10.21
• Towards Efficient Cross-modal Visual Textual Retrieval Using Transformer-encoder Deep Features (2021)6.34
• VITR: Augmenting Vision Transformers With Relation-focused Learning For Cross-modal Information Retrieval (2023)4.52
• Contra: (con)text (tra)nsformer For Cross-modal Video Retrieval (2022)2.26