Thank you for your constructive comments and suggestions! Please find our responses below.

Q1. The authors missed important previous works.

A1. Many thanks for your suggestion. CounterAnimal utilizes high-quality data available on the internet, whereas Terralcognita relies on camera trapping data, which are not as widely spread. We believe CounterAnimal is preferred over Terralcognita as CounterAnimal much align with the training / deployment setups of CLIPs. Moreover, to compare with ImageNet models, we should ensure that we align the label space of ImageNet, satisfied by CounterAnimal but not by Terralcognita. WaterBird also has such a problem, and is limited to simple binary classification. For the Background Challenge, they study the robustness from another perspective, i.e., w/ or w/o the background. However, to find spurious features within CLIP datasets, we need to compare model performance across different backgrounds. At last, SPIRE masks a part of objects from the original image, which may introduce new factors to degenerate model performance.

Q2. The naming of the common vs counter groups is misleading.

A2. Using accuracy instead of frequency makes the data collection procedure more effective in finding spurious features that make CLIP models fail. We will follow your suggestion and change the names of groups.

Q3. The terms "effective robustness" and "robustness" are used interchangeably which is wrong and confusing.

A3. We agree that we need to clarify the term “robustness” in Section 3. We further depict the lines of effective robustness, where we found that the conclusions remain the same. However, the improvement from increasing model scales and improving data quality is much higher than simply scaling up the datasets. We will add the related discussion in our revision.

Q4. Due to methodology issues, results on CLIP models cannot be compared to results on ImageNet models (or other advanced LVLMs).

A4. In general, we do not intend to claim that ImageNet models are generally more robust to spurious features. Instead, our goal is to emphasize that spurious features within CounterAnimal may not be as influential for ImageNet models, supporting that CounterAnimal captures spurious features within CLIPs, underscoring the significance of CounerAnimal. For previous works, they assess CLIP robustness using OOD datasets primarily for ImageNet datasets. Such a tendency may not fully reflect the CLIP robustness, as spurious features within ImageNet datasets may not be learned by CLIP models. Our CounterAnimal fills this gap, offering a more comprehensive perspectives when studying CLIP robustness.

Q5. Did the authors consider this possibility, i.e. that the drop in performance could be explained by other differences in the images from the respective domains?

A5. we have tried our best to do the quality control, and get rid of the influence of other factors. We also evaluate some potential confounders like gestures, as in the response to Reviewer iLSC, and find that it is relatively equally distributed across groups. Please kindly let us know if you feel there is another potential factor that may affect the performance.

Q6. Different test sets have different numbers of test images. ImageNet Val has 50k images for example. In what sense are the presented numbers standard?

A6. We aim at clarifying that our CounterAnimal (about 13K) aligns with the standard size as an evaluation dataset, resembling the scales of other many popular OOD evaluation datasets such as [1-2]. We will make our discussion clearer in our revision.

[1] The many faces of robustness: A critical analysis of out-of-distribution generalization.

[2] Do ImageNet Classifier Generalize to ImageNet?

Q7. Unclear description in the dataset construction procedure in Section 2.1.

A7. In data curation, feature noise refers to cases where some pixels are disrupted or missing. Clarity issues refer to cases where animal objects are largely occluded by backgrounds or other irrelevant objects. It also includes cases where animal objects do not occupy most of pictures. In background labeling, there are two cases where one image can be assigned more than one background label, i.e., some backgrounds can be confusing and other may possess more than one background. Due to space limit, we will further clarify our data construction procedure in our revision.

Q8. Further, I imagine animals occur in all kinds of environments, but there are only two backgrounds for each animal. Were the other images also discarded?

A8. Yes, we only preserve two groups for each animal, aiming to better capture spurious features captured of CLIPs. In Sec 3.1, we further show that the identified spurious features are general across different CLIP setups, justifying that these spurious features will not be largely bias towards a particular CLIP checkpoint.

Q9. “Therefore, we conclude that our CounterAnimal dataset possesses some realistic shifts that are generally contained in large-scale pre-training data, regardless of backbones.” This conclusion cannot be drawn from this experiment.

A9. We have shown that using different backbones will always lead to poor performance on the counter group in Fig 7. We will make our discussion clearer in our revision. Thank you for your suggestion.

Q10. The proposed experiment is very similar to the well-known ShiftMNIST [D] or ColoredMNIST [E] datasets, which test the influence of spurious correlations.

A10. The experiments are presented to echo our theoretical analysis that CLIP learns to align spurious features with object captions. For the legends in Fig. 11, “supervised” refers to the results of supervised trained models, while “obj” and “objbkg” refer to use different prompts to fine-tune CLIPs. We will add more discussion about the table legends in Fig. 11.

Q11. Typos and grammar issues.

A11. Thanks for pointing out our typos. We will correct them in our revision.

Thank you for your constructive comments and suggestions! Please find our responses below.

Q1. The proposed dataset is not sufficiently robust to analyze the influence of spurious bias, as this is not the only difference between the common and counter datasets.

A1. In our study, we primarily focus on the real-world spurious bias caused by backgrounds, while we also try our best to avoid the influence of other factors. As mentioned in Sec 2.1, during data curation, we took measures to control the influence of other covariates, e.g., ensuring that the main objects are in the major position. To further understand whether other factors such as gestures will introduce unintentional bias into the dataset, we count the proportion of different gestures within the class of Ice Bear. The results are listed as follows, which do not have an obvious difference in the distribution of gestures between common and counter scenarios. We will add the related discussion and explore other unintentional factors that may affect model predictions in our revision.

Q2. Current experiments cannot conclusively show that ImageNet models generalize better than CLIP.

A2. We apologize for any confusion in our description. In general, our conclusion that ImageNet models generalize better than CLIP models is limited to CounterAnimal. This is a preferred observation as it aligns with our main goal of crafting a dataset that characterizes the spurious features that may widely exist in CLIP setups. Prior to our work, there is no specialized benchmark curated for CLIPs to study the robustness of CLIPs to the real-world spurious correlations. Instead, previous works predominantly focus on the spurious features in ImageNets. Studying the OOD robustness of CLIPs on ImageNet specialized benchmarks is biased and may bring illusions about the robustness of CLIPs.

Q3. Prior studies [1,2,3,4,5,6] have proposed datasets specifically to analyze the influence of background, which are not discussed in this work. These datasets can be used for CLIP evaluation as they do not overlap with the CLIP training set. Additionally, creating datasets based on model accuracy in this work is similar to the approach in [6].

A3. Many thanks for your suggested paper. Most of these suggested papers have shown that CLIP models are more robust to the associated spurious features than ImageNet models, indicating that they did not intend to capture spurious features within CLIP setups. The CounterAnimal remains the first dataset that can find some spurious features that are predominant for the CLIP setups while might not be so strong for the ImageNet benchmarks, which is the uniqueness of our paper.

Thanks to the authors for providing the rebuttals, which partially addressed my concerns.

• Regarding Q2, I appreciate the adjustment to emphasize the spurious features found in CLIP setups. However, I did not find any discussion related to my original second point in Q2, which is quoted below:

  The accuracy drop from common to counter group can be greatly influenced by the model used to divide the common and counter dataset. Using the combined proposed common and counter dataset, a new Common' and Counter' dataset can be created based on the accuracy of ImageNet models. What is the impact of this dataset division on the accuracy drop for different models?

• Regarding Q2, could you please clarify the following point? For example, I would appreciate some examples of how prior work has focused on the spurious features in ImageNets.

  Prior to our work, there is no specialized benchmark curated for CLIPs to study the robustness of CLIPs to the real-world spurious correlations. Instead, previous works predominantly focus on the spurious features in ImageNets.

• Regarding Q3, the difference between this paper and prior studies remains unclear, which also affects the clarity of the paper's contribution. As also noted by reviewers KCA4 and BaJ3, the dataset creation makes it unsurprising that ImageNet models show a smaller gap between the two subsets. Therefore, it is confusing to claim that:

  The CounterAnimal remains the first dataset that can identify some spurious features predominant in CLIP setups while these features might not be as strong for the ImageNet benchmarks, which is the uniqueness of our paper.

• Regarding Q3. Prior studies have pointed out the spurious bias of the CLIP model about backgrounds, Therefore, a direct way to find CLIP-specific spurious bias is by splitting prior test set into two datasets—Counter and Common—based on CLIP accuracy. Could you further clarify how the proposed dataset differs from splitting prior test sets?

Thanks a lot to the authors for the detailed response and new experiments. I appreciate the changes to the title and the additional experiments, which have improved this work a lot in the rebuttal. I also appreciate the contribution of creating a real-world dataset about backgrounds. Considering all the comments and discussions, I am raising my rating to 5.

I share Reviewer 2MXa's concern that the paper needs significant revisions due to the shift in title and focus. I want to emphasize the modifications in the experimental results under the new shift. The original paper takes test accuracy as main experimental results to support the old title. However, test accuracy is not solid enough to be the main results for the new title as they are heavily influenced by the model used for dataset splitting. More analysis and results on spurious features are needed.

Thanks again to the authors for their efforts during the rebuttal and discussion.

Thank you for your constructive comments and suggestions! Please find our responses below.

Q1. Biased dataset: The dataset is split into common and counter groups using a CLIP model. Therefore, by construction, the CLIP models will perform poorly, and it is no surprise that the ImageNet-trained models do a bit better. One could construct a split of this dataset where ImageNet models do better than the CLIP models. If the dataset was collected in a model agnostic way then the conclusions could potentially be more interesting.

A1. We need to clarify that, one of the major objectives of this work is to construct a benchmark specifically to reflect the spurious correlations learned by CLIPs. To characterize the spurious features captured by CLIP, it is reasonable to use a CLIP model to curate the data. Our experiments in Section 3.1 justify that spurious features captured by CounterAnimal are general across different CLIP setups, and our experiments in Section 3.3 verify that the spurious features within CounterAnimal may not be so influential for ImageNet benchmarks. These results justify that our crafted CounterAnimal satisfies our original goal.

Furthermore, we would like to note that, previous comparisons between CLIPs and ImageNet models on ImageNet variant test sets are also biased. However, biases are not entirely bad. The past few years witnessed a lot of developments built upon the ImageNet variant test sets. Therefore, our benchmark share the same goal to provide a testbed for developing more advanced CLIP and vision-language models.

Q2. On the premise: Are there useful tasks/evaluations where ImageNet models are preferred over CLIP models? That is an interesting open question. Therefore the primary premise of the work seems erroneous. There is some value in the other evaluations so maybe the paper could be rewritten by positioning things differently.

A2. As explained in A1, our benchmark indeed captures spurious features learned by CLIP models. Future works can be developed upon our benchmark to mitigate the spurious correlations learned by CLIP models, similar to those built upon ImageNet variant testsets.

Moreover, we need to clarify that the comparison between ImageNet and CLIP models is not to argue which model is the best universally. Rather, we would like to highlight the biases existing in previous evaluations of CLIPs’ OOD robustness using ImageNet variant testsets. Benchmarking models with improper test sets would cause illusions that CLIP models seem to resolve the spurious correlations, especially compared with ImageNet models. However, the experiments with CounterAnimals provide a sober look of the vulnerability of CLIP models to spurious correlations, and provide a platform for future developments of more advanced and robust CLIP and vision-language models.

Finally, we understand that our original title may bring unnecessary misunderstandings about our work. We thus propose to revise it to A Sober Look at the Robustness of CLIPs to Spurious Features to more precisely reflect the contents of this work. Please kindly let us know if you have better suggestions to resolve the misunderstandings!

Q3. Lack of novelty: Keeping the primary result aside, other conclusions like that better datasets or model sizes improve robustness are not new.

A3. As clarified in A1 and A2, CounterAnimal is more suitable than ImageNet variant test sets to benchmark the OOD robustness of CLIP models. As the main comparison results between CLIPs and ImageNet models already differs from the previous studies, extending the benchmarking of CLIP models to more variants is necessary to verify the conclusions of previous works. We do not claim we are the first to discover those findings, rather, we are verifying those findings in order to provide insights of developing more robust CLIP and vision-language models.

Thank you for your response and the new title—it’s definitely an improvement. I believe this paper warrants acceptance, but additional revisions are necessary. The paper would benefit from aligning more closely with the direction of Hendrycks et al. (2019) in terms of the motivation. Specifically, the ImageNet/ResNet comparison needs to be reframed, as its current presentation is potentially misleading.

One potential experiment that could add value is identifying a common/counter split of the same ~13K dataset for ImageNet-trained models. Since you have the classes and backgrounds marked, it could be interesting to compare the differences between CLIP-based splits and ResNet-based splits. This comparison might reveals insights into the nature of spurious correlations in these two models. Incorporating these experiments and suggested changes could significantly enhance the manuscript. Hoping that the authors could make these changes, I am increasing my score.

Hendrycks et al. (2019) Natural Adversarial Examples

Thank you for your constructive comments and suggestions! Please find our responses below.

Q1. I think the claim is somewhat "obvious": there exists a relatively strong correlation between the object captions and the parts of image backgrounds, CLIP will learn to align the backgrounds, i.e., spurious features. If the training dataset contains many examples of spurious correlations, then models will tend to be biased.

A1. We need to clarify that, our theoretical analysis has unique and significant values for its implications to both theory and practice:

• From the theoretical perspective, it complements the literature of theoretical studies about CLIPs. Prior to this work, previous works such as [1,2] mainly focus on developing theories and experiments justifying the superior OOD generalization capabilities of CLIPs in ImageNet variant tests. To the best of our knowledge, our theory is the first to provably demonstrate the drawbacks of CLIPs in OOD generalization, providing the foundation for future developments tackling the issue.

• As a direct implication, the theory shows that CLIP training objectives can not offer additional robustness against spurious correlations. Consequently, the biases in large-scale multimodal training data such as LAION-5B is the major source for the spurious correlations learned by CLIPs. However, prior to our work, there is no proper benchmark capturing the spurious correlations for large-scale multimodal training data.

[1] Identifiability Results for Multimodal Contrastive Learning, ICLR’23.

[2] Does CLIP’s generalization performance mainly stem from high train-test similarity? ICLR’24.

Q2. I am curious about why ImageNet models may not be so influenced by the spurious bias in CounterAnimal. Is this because the ImageNet training set does not have too many spurious correlation examples? Or ImageNet has a spurious bias but such bias is different from the one in CounterAnimal? Please provide a discussion or share some insights on this question.

A2. Thank you for the insightful question. First, we need to clarify that, ImageNet models still learn the spurious correlations as evidenced by the performance gaps in CounterAnimal benchmark. One reason for the lower performance gaps of ImageNet models is that, CounterAnimal is specifically designed to reflect the biases of the CLIP pretrained datasets, similar to ImageNet variant test sets which are designed for ImageNets[3,4]. A key takeaway for the phenomenon is that, we need to carefully choose suitable benchmarks to evaluate the OOD robustness of different models.

[3] Do ImageNet Classifiers Generalize to ImageNet? ICML’19.

[4] Natural Adversarial Examples, CVPR’21.

Q3. This paper adopts absolute performance drop in Section 3.3. Such a metric may not be so robust. For example, model A drops from 40 to 39, and model B drops from 90 to 89. They drop the same but the I would say model B is better. Please comment on this and discuss the metric of absolute performance drop.

A3. Thank you for the suggestion. We further depict the lines of effective robustness, where we found that the conclusions remain the same. However, the improvement from increasing model scales and improving data quality is much higher than simply scaling up the datasets.

Q4. While I appreciate this work includes results on ColoredCOO and ColoredMINIST, some other spurious correlation benchmarks (e.g., WaterBirds) would be greater if they were also included.

A4. We would like to note that the WaterBirds benchmark is similar to ColoredCOO, that synthesizes unnatural images by composing birds and different image backgrounds. Essentially it can be considered as a binary classification variant of ColoredCOCO and ColoredMNIST, hence the results of ColoredCOO and ColoredMNIST also generalizes to WaterBirds. In addition, we need to clarify that the main focus of this work is to identify natural spurious correlations learned by CLIPs instead of the synthetic ones. CounterAnimal essentially captures the desired real-world spurious features, differing from previous synthetic benchmarks like WaterBirds.

Q5. The dataset is proposed, so a discussion on the potential bias/ privacy is needed.

A5. As discussed in question 12 of the checklist (see Appendix), we use the publicly available data from the internet, following the CC BY-NC license, which permits scientific use. We have revised the Broader Impacts in our manuscript to include a discussion regarding the issue.

Dear Authors,

Thank you for providing the response. I share the concern about using a CLIP model to curate the data. The newly added CounterAnimal-i dataset is helpful, but it could introduce differences in some of the observations made in the submission. For example, it raises questions about whether the title, “Do CLIP Models Always Generalize Better than ImageNet Models?” can still be fully addressed.

The authors now emphasize that “the focus of this work is to construct a test set for studying the robustness of CLIP models against real-world spurious features.” While this is a valid focus, it necessitates significant revisions to reflect this shift. For instance, the abstract and introduction currently do not align with this focus.

Given these points, I am actually in the boderline case. I would like to point out that the motivation behind the work is always good—whether it is to answer the title’s question or to study the real-world spurious features of CLIP models.

However, I would like to draw the Area Chairs’ attention to the fact that this submission needs significant modification, and some observations should be carefully adjusted based on the newly curated dataset. The presentation also needs to be revised to align with the goal of studying real-world spurious features of CLIP models.

Best regards,

Reviewer 2MXa

Dear Reviewer 2MXa,

Thank you for engaging in the discussion and acknowledging our motivation. We need to clarify that as our focus is to study the robustness of CLIP models, the additional experiments do not necessarily introduce significant modifications.

Meanwhile, to avoid any potential misunderstandings, we have changed the title to A Sober Look at the Robustness of CLIPs to Spurious Features to more precisely reflect the research problem that our original abstract and introduction are addressing.

To accommodate the concern of adopting CLIP models to curate the splits, we have revised our abstract and the introduction to more align with the corresponding work for ImageNet [1], which curates an OOD test set ImageNet-a according to the accuracy of ImageNet models. We have revised our manuscript to clarify that it is a common practice in the literature to curate the OOD test set to evaluate the robustness of neural networks. The revisions are also acknowledged and suggested by Reviewer KCA4.

Please kindly let us know if you still feel any additional revisions are needed. We would sincerely appreciate it if you could jointly take our clarifications above when making the final evaluation of this work.

Following the last response, we present the detailed results of CLIP and ImageNet models on CounterAnimal-i in this comment due to the character limits.