CLAY: Conditional Visual Similarity Modulation In Vision-language Embedding Space

Abstract

Human perception of visual similarity is inherently adaptive and subjective, depending on the users' interests and focus. However, most image retrieval systems fail to reflect this flexibility, relying on a fixed, monolithic metric that cannot incorporate multiple conditions simultaneously. To address this, we propose CLAY, an adaptive similarity computation method that reframes the embedding space of pretrained Vision-Language Models (VLMs) as a text-conditional similarity space without additional training. This design separates the textual conditioning process and visual feature extraction, allowing highly efficient and multi-conditioned retrieval with fixed visual embeddings. We also construct a synthetic evaluation dataset CLAY-EVAL, for comprehensive assessment under diverse conditioned retrieval settings. Experiments on standard datasets and our proposed dataset show that CLAY achieves high retrieval accuracy and notable computational efficiency compared to previous works.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keylim2026clay

Related papers

• Visual-text Cross Alignment: Refining The Similarity Score In Vision-language Models (2024)0.00
• Calibclip: Contextual Calibration Of Dominant Semantics For Text-driven Image Retrieval (2025)0.00
• VL-CLIP: Enhancing Multimodal Recommendations Via Visual Grounding And Llm-augmented CLIP Embeddings (2025)2.26
• Finetuning CLIP To Reason About Pairwise Differences (2024)0.00
• Evdclip: Improving Vision-language Retrieval With Entity Visual Descriptions From Large Language Models (2025)0.00
• Equivariant Similarity For Vision-language Foundation Models (2023)13.78
• Context-adaptive Multi-prompt Embedding With Large Language Models For Vision-language Alignment (2025)0.00
• Probvlm: Probabilistic Adapter For Frozen Vision-language Models (2023)13.41