Cross-media Similarity Metric Learning With Unified Deep Networks

Abstract

As a highlighting research topic in the multimedia area, cross-media retrieval aims to capture the complex correlations among multiple media types. Learning better shared representation and distance metric for multimedia data is important to boost the cross-media retrieval. Motivated by the strong ability of deep neural network in feature representation and comparison functions learning, we propose the Unified Network for Cross-media Similarity Metric (UNCSM) to associate cross-media shared representation learning with distance metric in a unified framework. First, we design a two-pathway deep network pretrained with contrastive loss, and employ double triplet similarity loss for fine-tuning to learn the shared representation for each media type by modeling the relative semantic similarity. Second, the metric network is designed for effectively calculating the cross-media similarity of the shared representation, by modeling the pairwise similar and dissimilar constraints. Compared to t

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyqi2017cross

Related papers

• Modality-specific Cross-modal Similarity Measurement With Recurrent Attention Network (2017)16.23
• Adversarial Cross-modal Retrieval Via Learning And Transferring Single-modal Similarities (2019)8.60
• Cross-modal Deep Metric Learning With Multi-task Regularization (2017)9.03
• Deep Learning Techniques For Future Intelligent Cross-media Retrieval (2020)0.00
• Deep Unified Multimodal Embeddings For Understanding Both Content And Users In Social Media Networks (2019)0.00
• Learning Joint Embedding For Cross-modal Retrieval (2019)5.84
• Conditional Similarity Networks (2016)15.06
• Comprehensive Graph-conditional Similarity Preserving Network For Unsupervised Cross-modal Hashing (2020)3.14