SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP

Dear Reviewer vKXP

We sincerely appreciate Reviewer vKXP for your thorough review and positive recognition of our work. Your suggestions are invaluable for improving our submission. Please find our responses below:

S1. I will start off by saying that the lede of the paper was buried: section 5 as an experiment is astounding. I absolutely love the drag and drop replacement of this debiased model into an existing model. The idea that, 1. you don't have to retrain the text-to-image model and 2. that you don't have to retrain CLIP from scratch is incredibly compelling. I recommend the abstract/intro be rewritten to highlight this experiment.

We thank the reviewer for highlighting this strength and for the kind words about the experiment in Section 5. We agree that the "drag-and-drop" nature of SANER, which avoids retraining both the text-to-image model and CLIP itself, is a particularly compelling aspect of our work.

Based on your suggestion, we have revised the abstract and the final paragraph of the introduction to emphasize this experiment and the practical advantages of SANER's flexible integration. This adjustment will better highlight the significance of SANER’s contributions and ensure its strengths are clearly communicated from the outset.

Specifically, we have highlighted that SANER works as a lightweight, drop-in replacement for CLIP in existing vision-language systems without requiring:

• Retraining of the text-to-image models
• Retraining of CLIP itself
• Access to the original training data

W1. I don't have too many issues with the paper. One thing I would like to have seen is a more automated method for text neutralization. While word level ablation is intuitive and makes sense for the social bias angle, a less hardcoded approach seems like it would be needed for a general-purpose debiasing solution, especially for concepts that are hard to ablate at the word level.

We thank the reviewer for this thoughtful suggestion regarding automating text neutralization. We agree that while our current word-level approach is effective for well-defined social biases, it could be extended to handle more complex cases.

As a potential future direction, embedding-based neutralization or context-aware language models could be explored to automate this process. These methods would enable bias identification and neutralization at a semantic level, reducing reliance on predefined attribute word lists and making the debiasing process more robust and adaptable to diverse scenarios.

However, we opted for our current approach because it:

• Provides precise control over attribute neutralization,
• Ensures reliable and interpretable transformations and
• Maintains grammatical correctness.

We appreciate this valuable suggestion and have included it as a discussion point in the future extension section (Appendix F).

W2. There's a dataset called FACET that addresses some issues with PATA and FairFace. If there's time, it would be great to run experiments on it as well. This would cement the proposed method as useful and cutting edge in my opinion.

Thank you for the valuable suggestion. FACET shares similarities with PATA as a dataset of real-world, natural images but provides a broader range of annotations for protected attributes, such as hair color. To evaluate SANER's effectiveness on FACET, which differs in data distribution from FairFace and PATA, we conducted experiments focusing on gender bias. The results, shown in the table below (also added to Appendix B.6), demonstrate SANER's superior debiasing performance, further supporting its robustness across diverse dataset distributions. Due to the short rebuttal period, we plan to include experiments on other attributes, such as hair color, in the final manuscript upon acceptance.

Gender bias on FACET, evaluated by MaxSkew@1000.

Q1. Have you considered running experiments with LLaVA-like models? ... However, I realize this experiment may best be left for future work.

Thank you for the suggestion. Incorporating LLaVA-like models into our experiments is a great idea and could serve as a potential extension of SANER while implementing these experiments would require additional setup, such as introducing debiasing module for image encoder, which are outside the current scope of this paper. We agree that this is a promising direction for future work and have added it to the future extension section (Appendix F). Thank you for suggesting this impactful extension.

Dear Reviewer STrF,

We sincerely appreciate Reviewer STrF for recognizing the importance of our study. Your comments are very helpful in improving our submission. Please find our responses below:

W1. Although replacing the protected attribute words with an attribute-neutral word is a sound neutralization method, it requires a comprehensive list of the protected attributes (as listed in Appx. C). However, in real practice, and especially in non-binary cases, it will be very hard or nearly impossible to have a complete list. Further, since the "attribute annotation-free debiasing loss" relies on a set of attribute-specific descriptions, creating the attribute-specific descriptions set may lead to some concerns. For example, if the set is created by iterating all possible words and all possible groups (e.g., man, woman, male, female, he, she, etc), then the size of the set will be overlarge and increase the computation complexity. On the other hand, if only one word is selected for each group (e.g., man + female, woman + male), then how to select matching words from groups may also affect the debias performance. Also, is it possible that the generated description may have unmatched text (e.g., A woman/boy is eating salad) or text that has grammar errors (e.g., A he/she is eating salad, according to Appx. C). I would like to see more discussion on the text generation.

We thank the reviewer for these insightful questions about text processing challenges. You raise several valid points:

Comprehensive attribute list

Yes, creating a complete list is challenging, especially for non-binary cases However, compared to the existing works [Berg et al., 2022; Seth et al., 2023] which require the attribute annotation to each image in the entire dataset, SANER's flexibility allows easy extension by simply adding new terms to the attribute list without requiring dataset reannotation. Our current implementation uses common terms that cover most cases in practice

Attribute-specific description generation

To avoid computational complexity, we implement a systematic approach using predefined mapping dictionaries between gender terms (e.g., "woman"→"man", "she"→"he"). Our algorithm carefully identifies and replaces gender-specific tokens while preserving sentence structure, ensuring no ungrammatical combinations (like "A he/she") are generated. This ensures efficient and coherent text generation that maintains natural language patterns.

We have added these implementation details to Appendix A.

W2. The proposed pipeline tried to address the challenge by combining the debiasing loss with two extra regularization losses. The debiasing loss tries to place the adjusted feature of the attribute-neutralized text in the middle point of several features of the attribute-specific text, which seems to be fine. However, the regularization losses try to project the attribute-neuralized feature to both text and visual features of an attribute-specific feature, which seems strange. As demonstrated in Figure 2, the input to the "Text encoder" has been neutralized, and the attribute-specific information has been removed. Further, the debiasing layer is also trained to adjust the features to be further neutral. Thus, even there is a residual connection in the "Feature modification" process, the output feature should not contain any information regarding the specific attribute. Then, how is it possible that such a feature can be mapped to either text or visual features that contain specific attribute information to avoid "the loss of attribute information"? Is it possible that the "retained attribute information" is some biased information from the dataset that is implicitly polluted by the regularization loss?

We apologize for the potential confusion in Figure 2. For clarity: while the debiasing loss uses neutralized descriptions ($h(\xi(t))$, e.g., "A person is eating"), the regularization losses use the original descriptions ($h(t)$, e.g., "A woman is eating"). This design ensures that:

• The debiasing loss adjusts neutralized features to be equidistant from multiple attribute-specific features, ensuring neutrality.
• The regularization losses (reconstruction and contrastive losses) maintain CLIP's original capabilities by preserving alignment with unmodified attribute-specific descriptions, preventing the degradation of semantic and visual consistency.

This approach ensures that attribute-neutral features are not directly mapped to attribute-specific ones, but rather that their neutrality is preserved alongside the overall semantic integrity of the features. Consequently, the concern about the regularization losses introducing unintended bias does not apply to our implementation.

To address this confusion, we have revised Figure 2 to clearly distinguish the inputs for debiasing and regularization losses. Thank you for helping us identify this potential source of misunderstanding.

I have read the authors' responses. My concerns have been addressed to some extent, but I still have concerns about the comprehensive list of protected attributes, which can limit the application of the proposed method. I have raised my rating to 6 to reflect the fact that some of my concerns have been addressed. In my opinion, this is a borderline paper and I will humbly leave the decision to AC's discretion.

Dear Reviewer wrAD,

We sincerely appreciate your recognition of our model's effectiveness. Your suggestions are invaluable for improving our submission. Please find our responses below:

W1. Could the authors provide ARL results to verify the loss of attribute information? Besides, do other existing methods, such as Mapper and Prompt tuning-based debiasing, also have the lossing attribute information when debaising?

We thank the reviewer for this suggestion. While the original ARL paper lacked sufficient implementation details for exact reproduction, we implemented ARL based on the information provided in their paper. Our implementation shows that ARL indeed loses attribute information, achieving accuracy of 0.64/0.83 for female/male in the generation experiments, similar to projection-based debiasing (0.58/0.79 in Table 4).

Regarding other methods:

• Prompt tuning-based debiasing retains attribute information as it only modifies concept-specific prompts
• Mapper reduces attribute correlation with task labels but may preserve explicit attributes

Our experiments in Table 4 quantitatively demonstrate SANER's advantage in preserving explicit attributes (1.00/1.00 accuracy) while reducing unwanted bias.

W2. Although effective in debiasing CLIP, the proposed method appears ad-hoc for this specific task and lacks major technical contributions, as most components seem to be existing technologies or tricks...

While acknowledging the reviewer's perspective, we respectfully disagree with the assessment. SANER makes several significant contributions to the research community from diverse aspects, as confirmed by the other reviewers:

1. Annotation-free debiasing: As noted by Reviewer STrF, we solve "critical and interesting" challenges with a lightweight yet effective approach. Unlike previous methods requiring costly attribute annotations to datasets, our novel debiasing strategy achieves superior performance using only text transformations.
2. Selective debiasing: As highlighted by Reviewer igmm, our method demonstrates "immediate practical application" by preserving explicitly specified attributes while removing unwanted bias - a fundamental advancement over existing approaches.
3. Practical impact: Reviwer vKXP found our approach "astounding" and "incredibly compelling" because it works as a drop-in replacement without retraining large models and demonstrates effectiveness on real applications like Stable Diffusion.
4. Broader applicability: Our method enables the use of any image-text dataset rather than being limited to face-centric datasets, with experiments showing this diversity improves performance.

The technical novelty combined with practical impact and broader applicability makes SANER a significant contribution to debiasing vision-language models.

W3. Why does adding only contrastive losses have a significantly negative impact on FairFace performance (as shown in Table 7)?

We thank the reviewer for this observation about the ablation results. When using only the contrastive loss (without the reconstruction loss), MaxSkew scores on FairFace increase (18.0/19.4/21.7 vs. 8.9/14.5/7.7 with both losses), indicating less effective debiasing. This occurs because the contrastive loss alone focuses on maintaining image-text alignment but does not constrain the debiased features to remain close to the original CLIP features. As a result, the feature modifications may become suboptimal for bias removal. The reconstruction loss is crucial in ensuring the modified features preserve essential semantic information while effectively removing unwanted bias.

We have added this discussion in Appendix B.2.

W4. Is the proposed method only applicable to CLIP? Could the authors test other VLMs to demonstrate its broader effectiveness?

We thank the reviewer for this important question. SANER can be applied to any VLM with vision and language encoders, as our method only requires access to the model's text encoder output. To demonstrate this, we conducted additional experiments with BLIP, where SANER shows superior debiasing performance as shown below (also added to Appendix B.5).

Gender bias for BLIP, evaluated by MaxSkew@1000.

Please note that many studies [Berg et al., 2022; Seth et al., 2023; Chuang et al., 2023] prioritize CLIP due to its widespread use, leaving other VLMs as valuable future work.

Dear Reviewer igmm,

We sincerely appreciate the positive review and your recognition of our work’s contribution. Your comments are very helpful in enhancing the quality of our presentation. Please find the corresponding responses below:

W1. The attribute groups will be limited to a specific set of defined attributes, which need to be agreed upon and could be debatable, e.g. "pregnant" (If I understand list C in the appendix correctly). There may also be missed attributes. This does not seem like a big issue but could be one in an adversarial setting.

Q3. Conceptually, it seems hard to draw a line between terms related only to the gender and to the gender and a concept. Should de-biasing replace terms like pregnant?...

We thank the reviewer for this insightful observation. You raise a valid point - while we currently focus on neutralizing direct demographic descriptors (e.g., "woman" → "person"), the boundary between gender-specific terms and concept-related terms is indeed complex. Terms like "pregnant" inherently carry gender-related information but also convey essential state information that should be preserved. We agree this deserves deeper consideration, particularly in how debiasing methods can balance removing unwanted bias while preserving meaningful context.

While we have aimed to make the attribute list as comprehensive as possible based on prior work [Berg et al., 2022; Seth et al., 2023; Hirota et al., 2023], we acknowledge that the choice of attribute groups may be subjective and that certain attributes might be missed. However, in contrast to previous approaches that require extensive attribute labels for images in the dataset, our method is designed to be flexible and extensible, allowing the attribute list to be expanded or adjusted based on specific needs or ethical considerations of the application domain.

In Appendix F, we have added a discussion about this potential extension.

W2. In general, the benefits of not needing a annotated dataset with attributes seems to be bought by needing a general list of attributes. This may not be a weakness per se but it would be interesting to either read about it in the limitations or have the author comment on why this is not a limitation.

Thank you for your insightful comment. As Reviewer igmm mentioned, while it is true that SANER avoids the need for annotated datasets by relying on a general list of attributes, we believe this is not a significant limitation. Though careful consideration is required, creating attribute lists is a one-time effort requiring minimal resources, compared to the ongoing cost and complexity of annotating protected attributes for people in image datasets that often needs ethical review and domain expertise.

We have added this discussion to the limitations section. Thank you for helping us improve the paper's clarity.

W3. There are general implementation details about the used architecture but none about the training process...

Q7. How hard is it to re-train clip? ...

We thank the reviewer for pointing out the lack of training details. To clarify, SANER does not retrain CLIP itself; instead, we train a lightweight debiasing layer (a two-layer MLP with a hidden dimension of 128) while keeping CLIP's parameters frozen. We observed that this results in a stable training process.

Training specifics:

• Hardware: Single NVIDIA A100 GPU (40GB)
• Training time: ~5 hours for 5 epochs on 170,624 COCO image-caption pairs
• Optimizer: AdamW with learning rate 5×10⁻⁶
• Batch size: 128
• Loss weights: α=1.0, β=0.1, γ=0.0001 for debiasing, reconstruction, and contrastive losses

With these efficient training setting, the regularization losses help maintain CLIP's performance, achieving similar zero-shot classification accuracy (65.2% vs original 65.4% on ImageNet) with minimal computational overhead.

We have added these implementation details to Section A of the appendix. Thank you for helping us improve reproducibility.

W4. Comparison against other state of the art in this small field is limited but understandably so and the author's did a good job re-implementing relevant approaches.

We appreciate the reviewer's understanding regarding the limited comparisons. As noted, while some recent methods exist [Seth et al., 2023; Dehdashtian et al., 2024], insufficient implementation details prevented reliable reproduction. We focused on thorough comparisons with methods having public code [Berg et al., 2022; Chuang et al., 2023] to ensure fair evaluation across multiple tasks (retrieval and generation) and attributes (gender, age, race).