Fairclip: Social Bias Elimination Based On Attribute Prototype Learning And Representation Neutralization

Abstract

The Vision-Language Pre-training (VLP) models like CLIP have gained popularity in recent years. However, many works found that the social biases hidden in CLIP easily manifest in downstream tasks, especially in image retrieval, which can have harmful effects on human society. In this work, we propose FairCLIP to eliminate the social bias in CLIP-based image retrieval without damaging the retrieval performance achieving the compatibility between the debiasing effect and the retrieval performance. FairCLIP is divided into two steps: Attribute Prototype Learning (APL) and Representation Neutralization (RN). In the first step, we extract the concepts needed for debiasing in CLIP. We use the query with learnable word vector prefixes as the extraction structure. In the second step, we first divide the attributes into target and bias attributes. By analysis, we find that both attributes have an impact on the bias. Therefore, we try to eliminate the bias by using Re-Representation Matrix (RRM)

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keywang2022fairclip

Related papers

• CLIP Is Shortsighted: Paying Attention Beyond The First Sentence (2026)0.00
• Advancing Myopia To Holism: Fully Contrastive Language-image Pre-training (2024)0.00
• Koo-fu CLIP: Closed-form Adaptation Of Vision-language Models Via Fukunaga-koontz Linear Discriminant Analysis (2026)0.00
• CLIP The Bias: How Useful Is Balancing Data In Multimodal Learning? (2024)0.00
• Clip-lite: Information Efficient Visual Representation Learning With Language Supervision (2021)2.35
• Clip-vip: Adapting Pre-trained Image-text Model To Video-language Representation Alignment (2022)5.42
• Addressing Bias In Vlms For Glaucoma Detection Without Protected Attribute Supervision (2025)0.00
• Mitigating Test-time Bias For Fair Image Retrieval (2023)0.00