Multi-modal Reasoning Graph For Scene-text Based Fine-grained Image Classification And Retrieval
2020 · Andres Mafla, Sounak Dey, Ali Furkan Biten, et al.
Abstract
Scene text instances found in natural images carry explicit semantic information that can provide important cues to solve a wide array of computer vision problems. In this paper, we focus on leveraging multi-modal content in the form of visual and textual cues to tackle the task of fine-grained image classification and retrieval. First, we obtain the text instances from images by employing a text reading system. Then, we combine textual features with salient image regions to exploit the complementary information carried by the two sources. Specifically, we employ a Graph Convolutional Network to perform multi-modal reasoning and obtain relationship-enhanced features by learning a common semantic space between salient objects and text found in an image. By obtaining an enhanced set of visual and textual features, the proposed model greatly outperforms the previous state-of-the-art in two different tasks, fine-grained classification and image retrieval in the Con-Text and Drink Bottle da
Authors
(none)
Tags
Stats
Related papers
- Beyond Visual Semantics: Exploring The Role Of Scene Text In Image Understanding (2019)9.59
- Scene Graph Based Fusion Network For Image-text Retrieval (2023)4.52
- Visual Semantic Reasoning For Image-text Matching (2019)25.23
- Stacmr: Scene-text Aware Cross-modal Retrieval (2020)10.48
- A Deep Local And Global Scene-graph Matching For Image-text Retrieval (2021)10.74
- Fine-grained Image Classification And Retrieval By Combining Visual And Locally Pooled Textual Features (2020)10.48
- Scene Text Retrieval Via Joint Text Detection And Similarity Learning (2021)16.16
- Fine-grained Video-text Retrieval With Hierarchical Graph Reasoning (2020)18.27