Multimodal Prediction Based On Graph Representations

Abstract

This paper proposes a learning model, based on rank-fusion graphs, for general applicability in multimodal prediction tasks, such as multimodal regression and image classification. Rank-fusion graphs encode information from multiple descriptors and retrieval models, thus being able to capture underlying relationships between modalities, samples, and the collection itself. The solution is based on the encoding of multiple ranks for a query (or test sample), defined according to different criteria, into a graph. Later, we project the generated graph into an induced vector space, creating fusion vectors, targeting broader generality and efficiency. A fusion vector estimator is then built to infer whether a multimodal input object refers to a class or not. Our method is capable of promoting a fusion model better than early-fusion and late-fusion alternatives. Performed experiments in the context of multiple multimodal and visual datasets, as well as several descriptors and retrieval models

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydourado2019multimodal

Related papers

• Fusion Vectors: Embedding Graph Fusions For Efficient Unsupervised Rank Aggregation (2019)0.00
• Dense Multimodal Fusion For Hierarchically Joint Representation (2018)11.49
• Learning To Represent Image And Text With Denotation Graph (2020)7.81
• Unigraph2: Learning A Unified Embedding Space To Bind Multimodal Graphs (2025)6.77
• Exploiting "quantum-like Interference" In Decision Fusion For Ranking Multimodal Documents (2018)0.00
• Cross-modal Fusion Distillation For Fine-grained Sketch-based Image Retrieval (2022)2.68
• Joint Fusion And Encoding: Advancing Multimodal Retrieval From The Ground Up (2025)0.00
• Everything At Once -- Multi-modal Fusion Transformer For Video Retrieval (2021)15.78