Dense Multimodal Fusion For Hierarchically Joint Representation

Abstract

Multiple modalities can provide more valuable information than single one by describing the same contents in various ways. Hence, it is highly expected to learn effective joint representation by fusing the features of different modalities. However, previous methods mainly focus on fusing the shallow features or high-level representations generated by unimodal deep networks, which only capture part of the hierarchical correlations across modalities. In this paper, we propose to densely integrate the representations by greedily stacking multiple shared layers between different modality-specific networks, which is named as Dense Multimodal Fusion (DMF). The joint representations in different shared layers can capture the correlations in different levels, and the connection between shared layers also provides an efficient way to learn the dependence among hierarchical correlations. These two properties jointly contribute to the multiple learning paths in DMF, which results in faster conver

Stats

• citations33
• S2 citations—
• github stars0
• HF likes0
• heat score11.49
• arxiv keyhu2018dense

Related papers

• The More, The Merrier: Contrastive Fusion For Higher-order Multimodal Alignment (2025)0.00
• Modal-aware Features For Multimodal Hashing (2019)0.00
• Joint Fusion And Encoding: Advancing Multimodal Retrieval From The Ground Up (2025)0.00
• Multimodal Prediction Based On Graph Representations (2019)0.00
• Data-efficient Multimodal Fusion On A Single GPU (2023)10.00
• Fusion-supervised Deep Cross-modal Hashing (2019)8.60
• Everything At Once -- Multi-modal Fusion Transformer For Video Retrieval (2021)15.78
• Co-attendwg: Co-attentive Dimension-wise Gating And Expert Fusion For Multi-modal Offensive Content Detection (2025)3.58