Debiasing Vison-Language Models with Text-Only Training

1. Not ready for submission: less than 10 pages.
2. The writing is poor, many typos, like 'use using' in Line 186, 'sSO' in Line278.
3. The text generation and MTP are limited in novelty, although the effect seems to be okay.

1. The paper needs careful proofreading, some writing errors are as follows:

• demonstrate -> demonstrates, line 041;
• false attributes -> bias attributes, line 183;
• Both training and inference process is -> Both training and inference processes are, line 184;
• we use using -> we use, line 183;
• $C_b$ -> $C_B$, line 244;
• $\frac{<\cdot,\cdot>\tau}{\cdots}$ -> $\frac{<\cdot,\cdot>/\tau}{\cdots}$, Eq.2 and Eq.4.
• sSo -> So, line 277;

2. Eq.5 is confusing. If I understand correctly, $p(y=i,b=j|x)$ is a joint probability of the target class and bias attribute, and how to compute $\mathrm{max}_j\ p(y=i,b=j|x)$ firstly?
3. Given that the training of the TOD framework necessitates executing the forward process of the CLIP text encoder for both the prompts and input text, the authors are encouraged to evaluate its training efficiency (time per step, GPU memory usage) in comparison to baseline models.
4. The contribution of this paper is somewhat limited. As the core contribution of the paper, the text-as-image training paradigm has been previously proposed in earlier works [1, 2, 3]. Besides, LLM-based instruction-following text generation from categories is also not fresh [2, 4]. To highlight the contribution of this paper, the authors are advised to explore different instruction templates and more efficient ways of generating text in terms of debiasing the CLIP model. Furthermore, although CLIP has well aligned texts and images in a unified embedding space, the modality gap between them still objectively exists. Therefore, finding ways to overcome this modal gap and enhance the cross-modal transfer ability of the TOD model will make this work more solid.

[1] Texts as Images in Prompt Tuning for Multi-Label Image Recognition, CVPR 2023.

[2] Text as Image: Learning Transferable Adapter for Multi-Label Classification, arXiv 2023.

[3] TAI++: Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt, IJCAI 2024.

[4] Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image Recognition, ICML 2024.

• This paper claims to debias VLMs, but only evaluates on CLIP models in the experiments. Can the proposed method be applied to debias other VLMs as well?

• The proposed method involves prompt tuning, which could be costly. Could the authors provide a detailed time comparison with other baselines?

(a) Intuition behind the proposed approach

• The proposed approach - Text-only Debiasing (TOD), is based on the idea that the optimized learnable text prompts are directly applicable to the image modality due to the image and text modalities sharing a unified representation space. However, [R1] shows that the image and text embeddings of CLIP are located in two separate regions in the representation space.
• [R1] essentially contradicts one of the fundamental motivating ideas for TOD. This can imply one of the following - (i) TOD is impervious to the modality gap in CLIP or (ii) TOD somehow implicitly bridges the modality gap, which seems unlikely, based on the reviewer’s understanding since there is no training for the image encoder. Can the authors discuss the proposed method in the context of this paper and explain how TOD works despite this modality gap in CLIP?
• Additionally, since the whole proposed framework hinges on this property of CLIP, can the authors present simple motivating zero-shot experiments to show that the optimized prompts are directly applicable to the image encoder?

(b) Experiments and results

• Choice of bias attributes- In L373-375, the authors list gender bias, age bias, and wavy hair bias. Similarly, in L353-355, they consider chubby, wearing a hat, and age as various bias attributes. However, there is no discussion on these choices of bias attributes from CelebA. Is there a specific reason behind the choice of these attributes, or were they randomly chosen? The authors should consider presenting a comprehensive list of experiments in the supplementary on various bias attributes of CelebA to demonstrate the effectiveness of the approach.

• Unknown bias attributes- In L416-422, the authors claim that existing works such as Orth-Cali require knowledge of bias attributes, while TOD does not. However, the reviewer feels that the description of TOD does not reflect this. In this experiment, the authors select an attribute at random to serve as an auxiliary attribute based on which a balanced dataset is generated. Doesn’t this count as “requiring knowledge of bias attributes” similar to Orth-Cali? Orth-Proj / Orth-Cali use knowledge of the bias attribute to obtain a projected representation without the bias attribute, while TOD uses knowledge of the bias attribute to generate a balanced dataset. Essentially, both of these methods use information of the bias attribute in some way. Could the authors clarify how TOD does not require knowledge of the bias attribute, despite the discussion seemingly pointing the other way?

• Results in Fig. 4- The authors demonstrate in Fig.4 that text-only training can perform on par with image-based training. However, there is no clear explanation of the experimental setup of the balanced image training setting. Can the authors provide some discussion on the details of this setup?

(c) Minor writing issues

• There are formatting errors (Eg. Author contributions and Acknowledgments left unchanged from the template) and grammatical errors (Eg: L704-707) in the paper. Additionally, there is an error in the heading of Table 1, i.e., the top section of Table 1 should be “methods with image data”. The reviewer suggests that the authors go through the entire paper to rectify such issues.

(c) Missing references

• R1 - Liang, Victor Weixin, et al. "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning." NeurIPS 2022..
• R2 - Seth, Ashish, Mayur Hemani, and Chirag Agarwal. "Dear: Debiasing vision-language models with additive residuals." CVPR 2023.