Modal-aware Features For Multimodal Hashing

Abstract

Many retrieval applications can benefit from multiple modalities, e.g., text that contains images on Wikipedia, for which how to represent multimodal data is the critical component. Most deep multimodal learning methods typically involve two steps to construct the joint representations: 1) learning of multiple intermediate features, with each intermediate feature corresponding to a modality, using separate and independent deep models; 2) merging the intermediate features into a joint representation using a fusion strategy. However, in the first step, these intermediate features do not have previous knowledge of each other and cannot fully exploit the information contained in the other modalities. In this paper, we present a modal-aware operation as a generic building block to capture the non-linear dependences among the heterogeneous intermediate features that can learn the underlying correlation structures in other multimodal data as soon as possible. The modal-aware operation consist

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzeng2019modal

Related papers

• Transitive Hashing Network For Heterogeneous Multimedia Retrieval (2016)8.35
• Unsupervised Multi-modal Hashing For Cross-modal Retrieval (2019)8.35
• Fusion-supervised Deep Cross-modal Hashing (2019)8.60
• Weakly-paired Cross-modal Hashing (2019)0.00
• Hashgan:attention-aware Deep Adversarial Hashing For Cross Modal Retrieval (2017)15.34
• Multi-modal Mutual Information Maximization: A Novel Approach For Unsupervised Deep Cross-modal Hashing (2021)12.02
• Dense Multimodal Fusion For Hierarchically Joint Representation (2018)11.49
• MTFH: A Matrix Tri-factorization Hashing Framework For Efficient Cross-modal Retrieval (2018)16.88