Adversarially Robust CLIP Models Can Induce Better (robust) Perceptual Metrics

Abstract

Measuring perceptual similarity is a key tool in computer vision. In recent years perceptual metrics based on features extracted from neural networks with large and diverse training sets, e.g. CLIP, have become popular. At the same time, the metrics extracted from features of neural networks are not adversarially robust. In this paper we show that adversarially robust CLIP models, called R-CLIP\(_\textrm\{F\}\), obtained by unsupervised adversarial fine-tuning induce a better and adversarially robust perceptual metric that outperforms existing metrics in a zero-shot setting, and further matches the performance of state-of-the-art metrics while being robust after fine-tuning. Moreover, our perceptual metric achieves strong performance on related tasks such as robust image-to-image retrieval, which becomes especially relevant when applied to "Not Safe for Work" (NSFW) content detection and dataset filtering. While standard perceptual metrics can be easily attacked by a small perturbation

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keycroce2025adversarially

Related papers

• Lipsim: A Provably Robust Perceptual Similarity Metric (2023)2.76
• Robustness In Both Domains: CLIP Needs A Robust Text Encoder (2025)2.16
• Optimizing CLIP Models For Image Retrieval With Maintained Joint-embedding Alignment (2024)6.34
• Robust Cross-modal Representation Learning With Progressive Self-distillation (2022)12.33
• Benchmarking Robustness Of Contrastive Learning Models For Medical Image-report Retrieval (2025)0.00
• Safe-clip: Removing NSFW Concepts From Vision-and-language Models (2023)13.41
• Revisiting Relevance Feedback For Clip-based Interactive Image Retrieval (2024)4.52
• CIBR: Cross-modal Information Bottleneck Regularization For Robust CLIP Generalization (2025)4.52