Bendvlm: Test-time Debiasing Of Vision-language Embeddings

Abstract

Vision-language model (VLM) embeddings have been shown to encode biases present in their training data, such as societal biases that prescribe negative characteristics to members of various racial and gender identities. VLMs are being quickly adopted for a variety of tasks ranging from few-shot classification to text-guided image generation, making debiasing VLM embeddings crucial. Debiasing approaches that fine-tune the VLM often suffer from catastrophic forgetting. On the other hand, fine-tuning-free methods typically utilize a "one-size-fits-all" approach that assumes that correlation with the spurious attribute can be explained using a single linear direction across all possible inputs. In this work, we propose Bend-VLM, a nonlinear, fine-tuning-free approach for VLM embedding debiasing that tailors the debiasing operation to each unique input. This allows for a more flexible debiasing approach. Additionally, we do not require knowledge of the set of inputs a priori to inference ti

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keygerych2024bendvlm

Related papers

• Addressing Bias In Vlms For Glaucoma Detection Without Protected Attribute Supervision (2025)0.00
• Blind To Position, Biased In Language: Probing Mid-layer Representational Bias In Vision-language Encoders For Zero-shot Language-grounded Spatial Understanding (2025)0.00
• Infusing Fine-grained Visual Knowledge To Vision-language Models (2025)0.00
• Lost In Embeddings: Information Loss In Vision-language Models (2025)0.00
• Spacevlm: Sub-space Modeling Of Negation In Vision-language Models (2025)0.00
• Mitigating Test-time Bias For Fair Image Retrieval (2023)0.00
• Probvlm: Probabilistic Adapter For Frozen Vision-language Models (2023)13.41
• VLMAE: Vision-language Masked Autoencoder (2022)0.00