FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs

We thank the reviewer for their feedback. Below are our responses to individual questions.

W1 The authors did not compare their results with unimodal baselines, i.e., techniques that debias only the image/text representations of the VLM.

A1 We compared our method with various baselines, both with and without ground truth $Y$. Among those, Orth-Cali (Chuang et al., 2023) only debiases the VLM’s text representations, and Contrastive Adapter (Zhang & Ré, 2022) debiases only the VLM’s image representations.

Moreover, FairVLM itself can be adapted to debias only one of the modalities by setting $\tau_I$ or $\tau_T$ to zero. In either case, we will have a closed-form solution for the encoder parameters in a single shot and will not need to perform any alternating optimization.

Our motivation for debiasing text and image encoders stems from prior studies demonstrating bias in both image encoders [R1-R3] and text encoders [R4-R8]. Subsequently, to achieve our goal of proposing a method that can perform under different settings and conditions of debiasing, we made FairVLM controllable so that it can easily be adapted to an unimodal debiasing setting by only setting $\tau_I$ or $\tau_T$ to zero.

W2 While the authors argue that RKHS has nice universal approximation properties, it's unclear how and why they aid in debiasing the original representations.

A2 The key to the effectiveness of FairVLM is debiasing with HSIC type measure of dependence, which measures non-linear dependence. HSIC is typically estimated through RKHS. And, as we explain on page 4, under the Choice of Encoder paragraph, RKHS lends itself to closed-form solutions for the optimization problem, which is vital when optimizing competing objectives. SGD-type algorithms are not well suited in such cases, as is common in GANs and other adversarial learning methods. We repeat our description in the paper below for completeness:

“Our choice of RKHS is motivated by several reasons. Debiasing is inherently an optimization problem with multiple competing objectives. In such cases, optimization is the primary bottleneck rather than model expressivity. This was also observed in Sadeghi et al. (2022). The closed-form solution afforded by our approach mitigates the optimization challenges (Sec. 4.3 and App. A.6).”

Moreover, in Section 3.1 and Figure 4, we provide a geometric illustration of how FairVLM works. Using a universal RKHS like RBF kernel maps the data into infinite dimensional space where all attributes are perhaps amenable to disentangling through linear mappings, thus allowing us to estimate the correct direction of $Y$ and $S$ labels. In this space, a linear correlation can capture all types of dependencies, i.e., all linear and non-linear dependencies.

If our responses addressed your initial comments, please consider raising the score.

We thank the reviewer for their feedback. In response to the question regarding the performance of the other zero-shot models on the CFD experiment, we now evaluated more than 100 publicly available VLMs in addition to CLIP on CFD. The results provided more insight into understanding the challenging nature of this particular CFD scenario. In the following, we respond to each comment.

W1 While one of core features of VLM is zero-shot classification, the suggested method still requires parameter tuning (RBF kernel parameter) and stacking test data, which could be a limitation of the method.

A1 Please recall that we consider two scenarios. One where we have ground truth $Y$ labels and the other without $Y$ labels. In both cases, ground truth $S$ labels are not available.

In both cases, we have a train-test split where we learn on the train split and evaluate on the test split. So, in both cases, there is no stacking of test data, and we can evaluate a single test sample at a time. We stack the train data in both cases and select the RBF kernel parameter through cross-validation. This experimental design choice was motivated by supervised baselines, which train with ground truth $Y$. We do not perform any online updates for FairVLM.

The reviewer’s comment suggests an alternative scenario where one learns FairVLM at test time and applies the learned model to the same test samples. In this scenario, yes, we have to stack the test data. However, even in this scenario, it is possible to pre-train FairVLM on a separate set of training samples (only images, without labels) obtained through other means and apply it to the test sample without training. For a completely novel task where samples for training (only images, without labels) cannot be obtained, we will have to stack test data for learning FairVLM. And, as we demonstrate, FairVLM can learn from limited data (e.g., CFD dataset with 597 samples).

Q1 Can this method be extended to a single point debiasing? (i.e. a single point inference for online prediction?)

A2 Yes, as we explained in response to the previous comment, we do not train at test time but instead train on a separate dataset. So at test time, we can debias a single point.

Q2 It looks like FairVLM sacrifices average scores more than other methods such as Contrastive Adapter in Table 2 w/ labels result. Furthermore, FairVLM works similarly in w/labels and w/o labels in CelebA. What’s a good interpretation on this?

A3 We note that the Contrastive Adapter is tailored for the supervised setting and employs a complex anchor-based loss function. So, it is critically dependent on access to ground-truth labels. Therefore, it cannot be easily adapted to scenarios without ground-truth labels. In contrast, FairVLM is designed to be flexible and equally applicable to settings with and without ground truth labels without customization. The anchor-based loss function in the Contrastive Adapter can be integrated into FairVLM to improve performance under supervised learning at the cost of losing the flexibility in adapting to all different scenarios in terms of label availability.

It is worth mentioning that even with this flexible approach, FairVLM can outperform Contrastive Adapter in both the CelebA and Waterbirds datasets of the CLIP ViT-L/14 model in terms of worst-group accuracy (WG) and Gap (Table 2) and also the Chicago Face Database (Table 3 (right)). Furthermore, since FairVLM can also mitigate the representation's unfairness and spurious correlations, it outperforms other baselines, including Contrastive Adapter, in fairness experiments such as CelebA for high cheekbones (Table 1).

In the experiments reported in Table 2 on the CelebA dataset, the target attribute ($Y$) is blonde hair or not. Since CLIP models are trained on large-scale and diverse datasets, their zero-shot prediction for the worst group on easy-to-predict attributes like blonde hair is closer to the average across all groups. So, there is no significant performance gap between learning FairVLM with and without labels.

To support this claim, we extract the zero-shot prediction accuracy of the target labels presented in Table 2 and report it below again (Table R1). We observe for CelebA that the gap between Avg. and the worst group is lower than Waterbirds, and the average accuracy is reasonably high (~87%). So FairVLM’s performance with and without labels is similar for CelebA, while the difference is larger for Waterbirds.

Table R1: $\hat{Y}$ zero-shot prediction accuracy by CLIP ViT-L/14

Thank you for the detailed clarification and updated results; most of my initial comments are nicely addressed. I have a follow-up comment on A2.

A2. Yes, as we explained in response to the previous comment, we do not train at test time but instead train on a separate dataset. So at test time, we can debias a single point.

My comment was about when such a separate dataset is unavailable, and we want to do zero-shot prediction for a single test data point. In my understanding, the answer is no. However, acquiring unlabeled dataset is typically not challenging, the proposed method remains valuable.

While I find the technical contribution of this paper to be sound and solid, I suggest the authors provide further clarification regarding the specific category and limitations of their proposed method. When true labels are available, the method can be seen as an adapter approach that uses kernel methods outlined in the paper. In this view, it might be inappropriate to label it as "zero-shot" prediction, since the method requires a sort of adapter training on the specific task. In the absence of true labels, the method can be viewed as a test-time adaptation using pseudo-labels. One of potential limitation is that applying it to a single or a few test data points can be challenging.

I have increased my score to 6.

We thank the reviewer for their feedback. Below are our responses to individual questions.

W1 The experiment results on the datasets (w/ labels) are not good enough.

A1 As pointed out by the reviewer as a strength, one of the main contributions of FairVLM is its ability to adapt to different settings to mitigate various types of biases. In this paper, we have conducted several experiments on FairVLM, including mitigating spurious correlation, mitigating intrinsic dependencies, and conducting experiments on datasets with limited training samples and both w/ and w/o ground-truth labels. In all of these settings, FairVLM either outperforms other baselines, e.g.,

• In Table 2, WG and Avg. increased for all experiments (except CelebA on CLIP ResNet-50) in w/o label settings,
• Chicago Face Dataset (CFD) in Table 3 (right)
• Reduce MaxSkew fairness metric for the FairFace dataset in Table 3 (left).
• Reduce the fairness metric EOD on the CelebA dataset in Table 1.
• Improve the WG accuracy and decrease the Gap for experiments on CLIP ViT-L/14 in Table 2.

Or it is second best in other experiments except for CelebA for CLIP ResNet-50 in w/ ground-truth labels.

Furthermore, FairVLM enjoys computational advantages for training (see Table 4). On the Waterbirds dataset, FairVLM is 40 times more efficient than Contrastive Adapter, which has some accuracy advantages over FairVLM in two of the experiments on CLIP ResNet-50 and is five times more efficient than the other baselines. On the CelebA dataset that contains more samples, FairVLM is more than ten times faster than the model with the best accuracy, DRF (Upsample), and about 100 times faster than the Contrastive Adapter, which is the second-best method in terms of accuracy in that specific setting. It is worth mentioning that since the FairVLM employs kernel methods in its encoders, it can be scaled to medium-sized datasets using Random Fourier Features [R1], while the other baselines are not efficient even for the relatively small-sized datasets (CelebA and Waterbirds), we evaluated.

In conclusion, based on the information provided above, we maintain a robust belief that suboptimal outcomes against specialized solutions occur in only two out of thirteen experiments. These results amply demonstrate the overall value and effectiveness of FairVLM's contributions.

W2 The pare of the method is too complex.

A2 Our solution employs kernel methods, which are part of the standard machine learning toolbox. Fig 4 in Section 3.1 provides a geometric illustration of FairVLM. Moreover, Fig. 2 provides an overview of the training and inference phases of the proposed method, and Fig. 3 explains our training algorithm as a figure.

If our responses addressed your initial comments, please consider raising the score.

References

[R1] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.

We thank the reviewer for their feedback. Below are our responses to individual questions.

W1 The terminology "VLM" is misused in this paper. VLM literally includes a vast area of models trained with vision and language. I feel that the title "FairVLM" and the terminology "VLM" are too much overclaimed, and a reader can misunderstand the focus of this paper.

A1 We thank the reviewer for detailing the generality of the term “VLM.” We narrowed the focus of our presentation to “debiasing CLIP embeddings for zero-shot predictions.” We renamed our approach to FairerCLIP and rephrased all instances of VLM with CLIP or zero-shot predictions with CLIP, as appropriate. We also changed the paper's title to “FairerCLIP: Mitigating Bias in CLIP Representations for Zero-Shot Prediction.”

To avoid confusing the other reviewers, we continue to use the term “FairVLM” in the rebuttal.

W2 I think this paper should tone down its contribution and focus more specifically. This paper only targets a feature refinement method top on the frozen CLIP model, and the comparison methods are also methods using frozen CLIP encoders.

A2 We have renamed our proposed method changed the title, and toned down our contributions and focus to “debiasing CLIP embeddings for zero-shot predictions.” We hope that our changes address the reviewer’s concerns.

While our approach operates on a frozen CLIP model, some of the baselines we compare against perform fine-tuning on at least one layer in CLIP if not all.

• ERM Linear Probe fine-tunes the last linear layer of CLIP.
• ERM Adapter adds new linear layers and fine-tunes them
• DFR fine-tunes the last layer of the CLIP

Despite not fine-tuning CLIP, FairVLM outperforms those that fine-tune CLIP, reiterating its effectiveness.

W3 Note that it is still a narrow topic in "de-biasing CLIP zero-shot prediction" because a number of works focus on the fine-tuning strategy [R10-12]. On the other hand, this paper relies on the pre-trained feature encoders that may weakens its contribution.

A3 Respectfully, we disagree with the reviewer that not fine-tuning the CLIP backbones weakens our contributions since. On the contrary, as we discuss below, operating directly on the features without fine-tuning is a more practical solution, especially under limited resources.

First, fine-tuning requires heavy computation and typically more data for fine-tuning compared to not resorting to fine-tuning. These may not be available in many practical scenarios, especially zero-shot prediction scenarios. Without sufficient care, fine-tuning may lead to severe over-fitting. Second, we already compare to baselines (ERM Linear Probe, DFR) that perform fine-tuning and demonstrate that FairVLM outperforms them significantly. Thirdly, FairVLM can be employed to debias feature representations in downstream tasks without access to the original images or model weights. Finally, employing fine-tuning along with FairVLM is undoubtedly feasible if required.

Based on the above, debasing frozen CLIP representations is not a narrow topic but has a wide range of practical application scenarios.

W4 there are a number of missing related works that should be discussed. For example, HSIC is a popular approach in de-biasing studies [R13, R14]. R13 directly optimizes the HSIC between features and sensitive attribute labels, similar to Dep(Z, Y) in the paper; R14 optimizes the HSIC between biased features and target features to avoid using sensitive attribute labels Y. If we extend our viewpoint to RHKS, there is work using MMD to achieve fairness [R15]. I omitted many HSIC-based regularization methods that could be related to this work, but if possible, it would be great to add more citations for methods using HSIC.

A4 Thank you for providing such a comprehensive list of suggestions for related work; we really appreciate it. We added the three suggested papers (R13-R15) to the related discussion on page 3 of the paper in the Choice of Dependence Measure paragraph. There are potentially other methods that use HSIC as a regularizer. However, due to page limits for the main paper, we could not add more of them at this time.

W5 in terms of employing pseudo-labels for the de-biasing optimization, I think this paper is also related to [R16]; where R16 is based on semi-supervised learning without the pseudo-label refinement process.

A5 From our understanding of R16, it has access to partially annotated sensitive attributes and estimates pseudo labels for the sensitive attribute for the rest of the training samples. In contrast, in our paper, the sensitive attributes are not available for FairVLM and other baselines, which makes our setting more challenging. We have added a citation to this paper on page 5 (colored in blue).