Modality-aware Triplet Hard Mining For Zero-shot Sketch-based Image Retrieval

Abstract

This paper tackles the Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) problem from the viewpoint of cross-modality metric learning. This task has two characteristics: 1) the zero-shot setting requires a metric space with good within-class compactness and the between-class discrepancy for recognizing the novel classes and 2) the sketch query and the photo gallery are in different modalities. The metric learning viewpoint benefits ZS-SBIR from two aspects. First, it facilitates improvement through recent good practices in deep metric learning (DML). By combining two fundamental learning approaches in DML, e.g., classification training and pairwise training, we set up a strong baseline for ZS-SBIR. Without bells and whistles, this baseline achieves competitive retrieval accuracy. Second, it provides an insight that properly suppressing the modality gap is critical. To this end, we design a novel method named Modality-Aware Triplet Hard Mining (MATHM). MATHM enhances the baseline with th

Stats

• citations0
• S2 citations—
• github stars7
• HF likes0
• heat score1.81
• arxiv keyhuang2021modality

Related papers

• Modality-aware Representation Learning For Zero-shot Sketch-based Image Retrieval (2024)8.60
• Relation-aware Meta-learning For Zero-shot Sketch-based Image Retrieval (2024)0.00
• Domain-smoothing Network For Zero-shot Sketch-based Image Retrieval (2021)13.92
• Semantic Adversarial Network For Zero-shot Sketch-based Image Retrieval (2019)10.74
• Zero-shot Everything Sketch-based Image Retrieval, And In Explainable Style (2023)16.67
• Ontology-aware Network For Zero-shot Sketch-based Image Retrieval (2023)6.34
• Cross-modal Subspace Learning For Fine-grained Sketch-based Image Retrieval (2017)13.34
• Doodle To Search: Practical Zero-shot Sketch-based Image Retrieval (2019)16.75