Mitigating Spurious Correlations in Zero-Shot Multimodal Models

We sincerely appreciate your positive feedback and thorough review. Your question holds great importance to us, and we would like to take this opportunity to address it.

(1) When RESNET-50 is used as the backbone, the proposed method performs poorly. Is there a way to solve the misalignment between text and image?

Q(1) For a fair comparison, we employed the same text prompts across models such as ViT and ResNet. However, we found that ResNet-50 shows potential misalignment between text and image embeddings.

We hypothesize that this misalignment stems from differences in expressive capabilities between the text and image modalities. Specifically, the ResNet-50 encoder shows reduced expressiveness in its representations.

To verify this hypothesis and address the misalignment, we conducted experiments using text prompts with reduced contextual information to emphasize the main object. This approach is inspired in the mechanics of text embeddings, which are computed via attention mechanisms. By reducing contextual information, attention is more focused on the main object, thereby reducing redundancy in the text embeddings. Such reduction is critical, as excessive information can overwhelm less expressive models like ResNet-50. We run experiments on the following spurious text prompts for both TIE and TIE*.

Table 1. Zeroshot classification results on Waterbirds with ResNet-50 for TIE.

Table 2. Zeroshot classification results on Waterbirds with ResNet-50 for TIE*.

From Tables 1 and 2, we observe that using shorter spurious text prompts improves the worst-group accuracy, significantly enhancing performance. This finding contradicts the recommended practices for text prompts in ViT-based models, as discussed in A1 of the general response. This further validates the existence of misalignment between text and image embeddings in ResNet-based models, which arises from differences in their underlying backbone architectures.

(2) How do we ensure the accuracy of the textual hints provided when using artificially identified fake features? Is there an evaluation mechanism to verify the effectiveness of these hints?

Q(2) The reliability of spurious text prompts is a significant challenge. The strength of our proposed method lies in its robustness, as it relies less on highly accurate spurious vectors compared to existing work. For further discussion on this method, we kindly refer our reviewer gGse to Section A1 of the general response.

Evaluating the effectiveness of text prompts is a critical aspect of multimodal model research. However, in our zero-shot problem setting, we do not have access to any target label information, making the evaluation of text prompts challenging. Nonetheless, we present our recommendations for handling spurious text prompts in our general response A1, which have been empirically validated. Given the less sensitive nature of our method, combined with best practices for constructing text prompts, it can effectively mitigate spurious correlations in the zero-shot classification of VLMs.

We sincerely appreciate your inspiring review and the valuable questions you've raised. We would like to take this opportunity to respond thoughtfully to each of them.

Weakness:

(1)Some notation need to be clarified

W(1) We will address this point in our responses within the question section.

(2) I think the TIE* procedure needs to be clarified more, do we need to know possible values of $t_a$ beforehand? this need to be made clearer in the early in the writing when TIE* is introduced.

W(2) Thank you for your suggestion, we will clarify this in our revision. $t_a$ is a predetermined spurious text prompt, with all possible values should be known beforehand. In our experiments, TIE* employs these spurious text prompts for zero-shot classification to assign pseudo-spurious labels. We will ensure this is explicitly stated in the revised manuscript.

Question

(1) What is $h_a$ in Theorem 1? is it image embeddings w/ spurious feature $a$ regardless of the label?

Q(1) Yes, $h_a$ is the image embedding with spurious feature $a$ regardless the label.

(2) What is the difference between $t_{spu}$ and $t_a$?

Q(2) Thank you for pointing out the interchangeable use of the subscripts $a$ and ${spu}$. Both notations are intended to denote the same concept: ${spu}$ refers to the spurious text prompt, and $a$ to the corresponding spurious label. We will ensure consistent notation in the revised version of our manuscript. Thank you!

(3): For TIE*, does it mean we need access to all possible spurious features descriptions? Not necessarily a drawback as mostly in datasets we know this beforehand.

Q(3) Yes, for TIE*, accessing all possible descriptions of spurious features is required. In experiments, the same spurious text prompts are used both to compute pseudo-spurious labels and to guide the translation of image embeddings.

Thank you for your considerate and encouraging review. We appreciate the opportunity to address your insightful question.

Weakness:

(1)TIE relies on accurate spurious prompts. The authors propose TIE* but its performance is not as good as TIE. Further investigation into the limitations of TIE* and potential improvements would help.

W(1) We kindly redirect our reviewer to Q1 in our response to Reviewer Y3Qc.

(2) The experiments focus on CLIP-based models. Exploring TIE/TIE* on other VLMs would be helpful.

W(2): Due to page constraints, we have included experiments with additional VLMs in the appendix of our original submission. In Section H.2, we conducted experiments using the ALIGN backbone [1] to perform zero-shot classification tasks. We observed that our method consistently achieves good performance in terms of worst group accuracy. This demonstrates the robustness of our method across different types of VLMs.

(3) nitpick: the capitalization of the term "TIE" implies its an acronym. If it is an acronym, it's unclear what it stands for.

W(3) Thank you for pointing this out. TIE stands for Text prompt based Image Embedding translation, which simplifies references in the paper.

Question:

(1) How sensitive is TIE to the choice of spurious prompts? Are there any guidelines for selecting effective prompts?

Q(1) We kindly redirect our reviewer to A1 section in general response.

(2) What are the failure cases of TIE* and how can automatic inference of spurious labels be improved?

Q(2) We kindly redirect our reviewer to A2 section in general response.

[1] Jia, Chao, et al. "Scaling up visual and vision-language representation learning with noisy text supervision." International conference on machine learning. PMLR, 2021.

(2) In large-scale datasets and tasks, how valid is the assumption of a single spurious feature for the entire dataset? How can spurious features be efficiently identified on a per-data-point basis?

Q2. This work builds upon existing research on spurious correlation [1,2,3,4]. Spurious correlation is a statistical concept that describes a relationship between two variables that appears to be related but is actually coincidental or influenced by an external variable [5]. Due to the statistical nature of spurious correlations, identifying spurious features on a per-data-point basis is challenging to achieve for now. This limitation is also noted in Section I of the appendix for further discussion.

For a comprehensive comparison, we conducted experiments across major benchmark datasets commonly used in spurious correlation studies. Our proposed method is capable of addressing multiple spurious correlations simultaneously by utilizing corresponding spurious text prompts, thus extending beyond a single-feature scenario. Additionally, our method supports multi-class spurious labels. For example, in the FMOW dataset, satellite images from diverse geographical regions show multi-class spurious features. Both TIE and TIE* have addressed multi-class spurious correlations in large-scale dataset studies.

[1] Wu, Shirley, et al. "Discover and cure: Concept-aware mitigation of spurious correlation." International Conference on Machine Learning. PMLR, 2023.

[2] Sagawa, Shiori, et al. "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization." arXiv preprint arXiv:1911.08731 (2019).

[3] Adila, Dyah, et al. "Zero-Shot Robustification of Zero-Shot Models." The Twelfth International Conference on Learning Representations. 2024.

[4] Kirichenko, Polina, Pavel Izmailov, and Andrew Gordon Wilson. "Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations." The Eleventh International Conference on Learning Representations.

[5] Ye, Wenqian, et al. "Spurious correlations in machine learning: A survey." arXiv preprint arXiv:2402.12715 (2024).

we sincerely appreciate your positive feedback and thorough review. We hope to take this opportunity to address your questions.

Weakness

(1) More in-depth discussion of the method is necessary (Why does it work? When does it fail? etc.)

W(1) [Why does it work?] Our work is grounded in a theoretical analysis which establishes that, under conditions of spurious correlations, the optimal translation operator for image embeddings aligns in the opposite direction of the spurious vector. Conceptually, this can be interpreted as neutralizing the influence of spurious features. In the domain of VLMs, the spurious vector can be effectively identified through the use of spurious text prompts. Building on this foundation, we propose TIE and TIE* as solutions to address spurious correlations in VLMs. Our method effectively integrates both image and text modalities, satisfying the requirements of VLMs and is supported by a solid theoretical foundation. These factors collectively provide a strong rationale for why our methods consistently achieve the best or second-best WGA performance across various datasets.

[When does it fail?] We kindly redirect Reviewer Y3Qc to the general response A2 for detailed information.

(2) English can be improved.

W(2) Thank you for your suggestion. We will refine our wording in the revised version.

Question

(1) Have you conducted an evaluation of the quality of the predicted pseudo-spurious labels on a validation set in order to analyze and bridge the accuracy gap between TIE and TIE*?

Q1. In Section 4.4, we investigated the accuracy of pseudo-spurious labels and their relationship with the worst group accuracy (WGA) of the zero-shot classification, as shown in Figure 4 on the right-hand side. We demonstrated that the accuracy of predicting spurious labels monotonically benefits the WGA in zero-shot classification. To further validate this relationship, we conducted experiments on the CelebA dataset, where we computed the accuracy for the spurious label 'male' using various models: CLIP ViT-B32, CLIP ViT-L 14, and CLIP ResNet-50. The results are shown below.

Table 1. Relationship between pseudo spurious label accuracy and performance Gap between TIE and TIE*.

This finding further validates that higher accuracy in predicting pseudo-spurious labels corresponds to a smaller gap between TIE and TIE*. This motivates our research into methods of enhancing TIE*'s performance. We propose three approaches to enhance performance: employing group-robust text prompts (as discussed in Section 4.3), using a small proportion of labeled data to accurately assign pseudo-spurious labels (as discussed in Section 4.4), and following text prompt guidelines to improve the performance of spurious text prompts (as discussed in A1 of the General Response). These three methods can effectively reduce the performance gap between TIE and TIE*.