Axiomatic Explanations For Visual Search, Retrieval, And Similarity Learning

Abstract

Visual search, recommendation, and contrastive similarity learning power technologies that impact billions of users worldwide. Modern model architectures can be complex and difficult to interpret, and there are several competing techniques one can use to explain a search engine's behavior. We show that the theory of fair credit assignment provides a \(\textit\{unique\}\) axiomatic solution that generalizes several existing recommendation- and metric-explainability techniques in the literature. Using this formalism, we show when existing approaches violate "fairness" and derive methods that sidestep these shortcomings and naturally handle counterfactual information. More specifically, we show existing approaches implicitly approximate second-order Shapley-Taylor indices and extend CAM, GradCAM, LIME, SHAP, SBSM, and other methods to search engines. These extensions can extract pairwise correspondences between images from trained \(\textit\{opaque-box\}\) models. We also introduce a fast

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhamilton2021axiomatic

Related papers

• Visual Similarity Attention (2019)0.00
• Rethinking Similarity Search: Embracing Smarter Mechanisms Over Smarter Data (2023)0.00
• Towards Visually Explaining Similarity Models (2020)0.00
• From Pixels To Purchase: Building And Evaluating A Taxonomy-decoupled Visual Search Engine For Home Goods E-commerce (2026)0.00
• Exploiting Local Indexing And Deep Feature Confidence Scores For Fast Image-to-video Search (2018)2.26
• Towards Axiomatic Explanations For Neural Ranking Models (2021)10.35
• Representation Learning For Efficient And Effective Similarity Search And Recommendation (2021)0.00
• Joint Modeling And Optimization Of Search And Recommendation (2018)0.00