Explaining Similarity In Vision-language Encoders With Weighted Banzhaf Interactions

Abstract

Language-image pre-training (LIP) enables the development of vision-language models capable of zero-shot classification, localization, multimodal retrieval, and semantic understanding. Various explanation methods have been proposed to visualize the importance of input image-text pairs on the model's similarity outputs. However, popular saliency maps are limited by capturing only first-order attributions, overlooking the complex cross-modal interactions intrinsic to such encoders. We introduce faithful interaction explanations of LIP models (FIxLIP) as a unified approach to decomposing the similarity in vision-language encoders. FIxLIP is rooted in game theory, where we analyze how using the weighted Banzhaf interaction index offers greater flexibility and improves computational efficiency over the Shapley interaction quantification framework. From a practical perspective, we propose how to naturally extend explanation evaluation metrics, such as the pointing game and area between the i

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keybaniecki2025explaining

Related papers

• Lightclip: Learning Multi-level Interaction For Lightweight Vision-language Models (2023)0.00
• Towards Visually Explaining Similarity Models (2020)0.00
• Equivariant Similarity For Vision-language Foundation Models (2023)13.78
• Blind To Position, Biased In Language: Probing Mid-layer Representational Bias In Vision-language Encoders For Zero-shot Language-grounded Spatial Understanding (2025)0.00
• Unified Loss Of Pair Similarity Optimization For Vision-language Retrieval (2022)0.00
• Structxlip: Enhancing Vision-language Models With Multimodal Structural Cues (2026)3.95
• Axiomatic Explanations For Visual Search, Retrieval, And Similarity Learning (2021)0.00
• Fuselip: Multimodal Embeddings Via Early Fusion Of Discrete Tokens (2025)0.00