Learning To Represent Image And Text With Denotation Graph

Abstract

Learning to fuse vision and language information and representing them is an important research problem with many applications. Recent progresses have leveraged the ideas of pre-training (from language modeling) and attention layers in Transformers to learn representation from datasets containing images aligned with linguistic expressions that describe the images. In this paper, we propose learning representations from a set of implied, visually grounded expressions between image and text, automatically mined from those datasets. In particular, we use denotation graphs to represent how specific concepts (such as sentences describing images) can be linked to abstract and generic concepts (such as short phrases) that are also visually grounded. This type of generic-to-specific relations can be discovered using linguistic analysis tools. We propose methods to incorporate such relations into learning representation. We show that state-of-the-art multimodal learning models can be further im

Stats

• citations10
• S2 citations—
• github stars0
• HF likes0
• heat score7.81
• arxiv keyzhang2020learning

Related papers

• Multimodal Prediction Based On Graph Representations (2019)0.00
• Unifying Vision-language Representation Space With Single-tower Transformer (2022)7.16
• Unigraph2: Learning A Unified Embedding Space To Bind Multimodal Graphs (2025)6.77
• Cross-modal Fusion Distillation For Fine-grained Sketch-based Image Retrieval (2022)2.68
• Generating Images With Multimodal Language Models (2023)6.77
• See, Hear, And Read: Deep Aligned Representations (2017)0.00
• Show, Translate And Tell (2019)4.52
• Fuselip: Multimodal Embeddings Via Early Fusion Of Discrete Tokens (2025)0.00