Adapt And Align To Improve Zero-shot Sketch-based Image Retrieval

Abstract

Zero-shot sketch-based image retrieval (ZS-SBIR) is challenging due to the cross-domain nature of sketches and photos, as well as the semantic gap between seen and unseen image distributions. Previous methods fine-tune pre-trained models with various side information and learning strategies to learn a compact feature space that is shared between the sketch and photo domains and bridges seen and unseen classes. However, these efforts are inadequate in adapting domains and transferring knowledge from seen to unseen classes. In this paper, we present an effective ``Adapt and Align'' approach to address the key challenges. Specifically, we insert simple and lightweight domain adapters to learn new abstract concepts of the sketch domain and improve cross-domain representation capabilities. Inspired by recent advances in image-text foundation models (e.g., CLIP) on zero-shot scenarios, we explicitly align the learned image embedding with a more semantic text embedding to achieve the desired

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keydong2023adapt

Related papers

• Semantic Adversarial Network For Zero-shot Sketch-based Image Retrieval (2019)10.74
• Bda-sketret: Bi-level Domain Adaptation For Zero-shot SBIR (2022)10.35
• An Efficient Framework For Zero-shot Sketch-based Image Retrieval (2021)13.65
• Stacked Semantic-guided Network For Zero-shot Sketch-based Image Retrieval (2019)0.00
• Symmetrical Bidirectional Knowledge Alignment For Zero-shot Sketch-based Image Retrieval (2023)4.52
• Modality-aware Representation Learning For Zero-shot Sketch-based Image Retrieval (2024)8.60
• CLIP For All Things Zero-shot Sketch-based Image Retrieval, Fine-grained Or Not (2023)15.54
• Relation-aware Meta-learning For Zero-shot Sketch-based Image Retrieval (2024)0.00