Hierarchy-based Image Embeddings For Semantic Image Retrieval

Abstract

Deep neural networks trained for classification have been found to learn powerful image representations, which are also often used for other tasks such as comparing images w.r.t. their visual similarity. However, visual similarity does not imply semantic similarity. In order to learn semantically discriminative features, we propose to map images onto class embeddings whose pair-wise dot products correspond to a measure of semantic similarity between classes. Such an embedding does not only improve image retrieval results, but could also facilitate integrating semantics for other tasks, e.g., novelty detection or few-shot learning. We introduce a deterministic algorithm for computing the class centroids directly based on prior world-knowledge encoded in a hierarchy of classes such as WordNet. Experiments on CIFAR-100, NABirds, and ImageNet show that our learned semantic image embeddings improve the semantic consistency of image retrieval results by a large margin.

Stats

• citations69
• S2 citations—
• github stars0
• HF likes0
• heat score13.84
• arxiv keybarz2018hierarchy

Related papers

• Integrating Visual And Semantic Similarity Using Hierarchies For Image Retrieval (2023)4.52
• Semantic Hierarchy Preserving Deep Hashing For Large-scale Image Retrieval (2019)5.24
• Hierarchy Neighborhood Discriminative Hashing For An Unified View Of Single-label And Multi-label Image Retrieval (2019)0.00
• Deep Hashing With Semantic Hash Centers For Image Retrieval (2025)2.26
• Learning To Embed Semantic Similarity For Joint Image-text Retrieval (2022)7.50
• Adaptive Semantic-visual Tree For Hierarchical Embeddings (2020)8.09
• Deep Class-wise Hashing: Semantics-preserving Hashing Via Class-wise Loss (2018)11.19
• SHREWD: Semantic Hierarchy-based Relational Embeddings For Weakly-supervised Deep Hashing (2019)0.00