Towards Visually Explaining Similarity Models

Abstract

We consider the problem of visually explaining similarity models, i.e., explaining why a model predicts two images to be similar in addition to producing a scalar score. While much recent work in visual model interpretability has focused on gradient-based attention, these methods rely on a classification module to generate visual explanations. Consequently, they cannot readily explain other kinds of models that do not use or need classification-like loss functions (e.g., similarity models trained with a metric learning loss). In this work, we bridge this crucial gap, presenting a method to generate gradient-based visual attention for image similarity predictors. By relying solely on the learned feature embedding, we show that our approach can be applied to any kind of CNN-based similarity architecture, an important step towards generic visual explainability. We show that our resulting attention maps serve more than just interpretability; they can be infused into the model learning proc

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzheng2020towards

Related papers

• Visual Similarity Attention (2019)0.00
• Visual Explanation For Deep Metric Learning (2019)14.36
• Visual Explanation Via Similar Feature Activation For Metric Learning (2025)0.00
• Attributable Visual Similarity Learning (2022)11.68
• Visualizing Deep Similarity Networks (2019)11.85
• Axiomatic Explanations For Visual Search, Retrieval, And Similarity Learning (2021)0.00
• Explaining Similarity In Vision-language Encoders With Weighted Banzhaf Interactions (2025)0.00
• Case-enhanced Vision Transformer: Improving Explanations Of Image Similarity With A Vit-based Similarity Metric (2024)0.00