Embedding Arithmetic Of Multimodal Queries For Image Retrieval

Abstract

Latent text representations exhibit geometric regularities, such as the famous analogy: queen is to king what woman is to man. Such structured semantic relations were not demonstrated on image representations. Recent works aiming at bridging this semantic gap embed images and text into a multimodal space, enabling the transfer of text-defined transformations to the image modality. We introduce the SIMAT dataset to evaluate the task of Image Retrieval with Multimodal queries. SIMAT contains 6k images and 18k textual transformation queries that aim at either replacing scene elements or changing pairwise relationships between scene elements. The goal is to retrieve an image consistent with the (source image, text transformation) query. We use an image/text matching oracle (OSCAR) to assess whether the image transformation is successful. The SIMAT dataset will be publicly available. We use SIMAT to evaluate the geometric properties of multimodal embedding spaces trained with an image/text

Stats

• citations15
• S2 citations—
• github stars0
• HF likes0
• heat score9.03
• arxiv keycouairon2021embedding

Related papers

• Deep Multimodal Image-text Embeddings For Automatic Cross-media Retrieval (2020)0.00
• Using Text To Teach Image Retrieval (2020)5.24
• Image Search Using Multilingual Texts: A Cross-modal Learning Approach Between Image And Text (2019)0.00
• Probabilistic Compositional Embeddings For Multimodal Image Retrieval (2022)13.80
• Telling The What While Pointing To The Where: Multimodal Queries For Image Retrieval (2021)10.07
• Bringing Multimodality To Amazon Visual Search System (2024)6.34
• Image-text Retrieval Via Preserving Main Semantics Of Vision (2023)10.22
• Revisiting Cross Modal Retrieval (2018)0.00