In Search of Forgotten Domain Generalization

Dear Reviewer 8EXX,
Thank you for acknowledging our paper’s strength, helpful comments, and feedback. We address each of your concerns below:

[Clarification of core insight]: Please note that before, numerous studies implicitly or explicitly assume that CLIP generalizes OOD due to inherent robustness or emergent properties of large training datasets. This is reflected in the following quotes:

• “zero-shot transfer is more an evaluation of CLIP’s robustness to distribution shift and domain generalization” [1]
• “several studies suggested that some Vision-Language Models (VLMs) such as the CLIPs, exhibit OoD generalization” [2]
• “VLP (Vision-Language Pretraining) models not only show good performance in-distribution (ID) but also generalize to out-of-distribution (OOD) data” [3]
• “models such as CLIP, ALIGN, and BASIC have recently demonstrated unprecedented robustness on a variety of natural distribution shifts” [4]
• “These models (CLIP, GLIP, etc.) have demonstrated enormous potential in, a wide range of downstream applications… especially for the out-of-distribution (OOD) samples” [5]
• “CLIP achieves OOD generalization in a task-agnostic way” [6].

This means that our paper challenges a common belief through rigorous experiments: When truly OOD test data is used—by removing test domain samples from training—CLIP fails to maintain robustness, revealing a critical flaw in standard evaluations. This insight highlights the need for rigorous OOD evaluations and tempers expectations about CLIP’s true generalization capabilities. Given this context, we hope you can see the novelty of our work. To better emphasize this core insight, we have revised the main text (Lines 80–88) and included a more detailed discussion (Sec. 6 first paragraph).

[Usefulness for subsequent researchers]: Our work not only provides key insights into dataset design and serves as a cautionary tale for OOD evaluations but also offers invaluable resources to the research community—a critical step toward improving model robustness. Specifically, we contribute:

• the LAION-Natural and LAION-Rendition datasets, each containing tens of millions of images that are rigorously OOD with respect to ImageNet and DomainNet test sets in terms of style
• domain classifier checkpoints
• domain classifier training data
• trained CLIP model checkpoints

To our knowledge, this is the first effort to deliver pretraining datasets of such scale and domain specificity, addressing a major gap in OOD generalization research. The value of these contributions is reflected in reviewer feedback, with reviewer HGzr describing our work as tackling an “extremely important problem” with “impactful findings,” “useful datasets,” and “useful methodology,” while reviewer Gz5f highlights that the paper “studies a valuable problem” with “sound analysis” to support its claims.

[Sensitivity to model architecture]: Prior work provides strong evidence that our data contamination findings and the optimal ratio findings transfer to other architectures and training methods, not just CLIP [4,7]. For instance, [4] “systematically ruled out the training set size, language supervision, and the contrastive loss function as explanations for the large robustness gains achieved by the CLIP models [...] CLIP’s robustness is dominated by the choice of the training distribution, with other factors playing a small or non-existent role.” They further show that models trained on identical data distributions but with varying loss functions (SimCLR+FT, CLIP, Supervised) or model architectures (different backbones, parameter sizes) all exhibit the same effective robustness. This suggests that our conclusions will hold broadly across model types. We have included a new paragraph in the discussion section to address this point (Sec. 6).

We hope that our detailed explanations have adequately addressed your questions and concerns raised. We sincerely hope this information offers a clearer understanding of our work, allowing you to increase your score.

[1] Radford et al, “Learning Transferable Visual Models From Natural Language Supervision”
[2] Abbasi et al, “Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models”
[3] Nguyen et al, “SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning”
[4] Fang et al, “Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)”
[5] Li et al. “Distilling Large Vision-Language Model with Out-of-Distribution Generalizability” (2023)
[6] Shu et al, “CLIPood: Generalizing CLIP to Out-of-Distributions”
[7] Miller et al. Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

Dear Reviewer Gz5f,
Thank you for your thoughtful feedback, insightful comments and questions, and for recognizing the strengths of our paper. Below, we address each of your concerns in detail:

[W1 - Improving the figure]: Based on your suggestion, we have improved the figure for better readability.

[W2 - Effectiveness of the Domain Classifier]: Thank you for raising this important point. The domain classifiers were trained, validated, and tested on randomly sampled subsets of LAION-200M (13K training, 3K validation, and 3K test images), ensuring no distribution shift between their training and evaluation data. While potential generalization error cannot be ruled out, we are confident in the classifier's effectiveness for the following reasons:

1. High Precision and Recall: We prioritized precision by thresholding the classifiers at 98% validation precision on the 3K LAION-200M validation set. The precision and recall metrics on the 3K LAION-200M test set, as well as on 1K samples each from ImageNet and DomainNet test sets, are presented in Tables 1 and 5. These high values demonstrate the classifiers’ reliability. Additionally, accuracy values without thresholding, reported in Tabs. 7 and 8 for LAION-200M and ImageNet/DomainNet sets, further affirm its robustness.

2. Visual Verification: Random samples from the LAION-Natural and LAION-Rendition datasets, derived from deploying the classifiers on LAION-200M, are visualized in Figs. 3, 21, and 22. These samples clearly align with the intended natural and rendition categories, underscoring the classifiers’ efficacy.

3. Empirical Validation via Core Results: Our core results (Sec. 4) show that training models exclusively on natural or rendition subsets leads to strong performance on the corresponding domain but weaker performance on the other. This outcome supports the accuracy of the domain classifiers in retrieving domain-specific samples.

We agree that ensembling different classifiers could further improve performance by leveraging diverse model biases. However, given the already strong performance of our current classifiers, we leave this as an avenue for future work. Finally, in response to your suggestion, we have added a discussion in Sec. A.9 summarizing these points, alongside details in Secs. 3.1, A.1.2, and A.1.5.

Dear authors,

Thanks for the comments and rebuttal. I believe all my concerns are addressed. I have been thinking about this topic for a while, which I termed "generalization is probably a hallucination during the era of the foundation model". One interesting point is whether the conclusion would also generalize to LLMs, which most people believe has achieved some sort of generalization.

Best

[W3 - Imperfect domain classifier]: We acknowledge the high percentage of images classified as ambiguous, caused by two design choices: Our goal was to isolate images that are strictly natural or strictly renditions. We, therefore, define as ambiguous any image containing stylized and natural elements, such as photos of tattoos, or objects with stylized logos (see Appx. A.1.1). Additionally, we prioritized high precision by deploying our classifier at 98% validation precision, which inevitably led to more false negatives labeled as ambiguous. Our conservative strategy ensures that the images identified as natural or renditions are highly reliable. Overall, the large number of samples our classifier can annotate for each domain and the existence of distinct datasets, such as ImageNet-Val and ImageNet-Sketch, support the validity of treating natural and rendition images as distinct data sources. Nevertheless, we agree that further exploring the characteristics and impact of borderline or ambiguous examples could provide valuable insights into generalization. We leave this exercise for future work.

[W4 - Lack of clarity]: We completely agree with your observation “if rendition data is present in the training data, then rendition data is no longer OOD with respect to the training distribution.” This is precisely the core point of our work. As referenced in [1–6], many works implicitly or explicitly assume that CLIP demonstrates OOD generalization. However, our findings challenge this assumption, showing that CLIP primarily achieves in-domain generalization due to domain contamination in its training data. To clarify this in the text, we have revised ambiguous statements like “domain contamination drives CLIP’s robustness,” to “CLIP’s apparent OOD robustness on standard OOD benchmarks like ImageNet-Sketch is often an artifact of overlapping domain data, rather than genuine OOD generalization.” This adjustment will ensure the language better conveys our central message.

[Q2 - Defining domain differences]: Thank you for highlighting this excellent reference. We intended to emphasize that measuring domain overlap can be challenging in practice, especially on web-scale data. We will cite the work by Ben-David et al. and revise the language to: "Domains, while often challenging to quantify in practice [7], emerge from collecting data from specific sources and conditions."

[Q3 - Annotation details]: Thank you for your suggestion. The details of the labeling procedure, including the annotators and quality assurance measures, are provided in Sec. 3.1, Appendix Sec. A.1.1, and A.1.4. In response to your recommendation, we have added further specifics. We appreciate your feedback in helping us improve the clarity and completeness of this section.

[Q4 - Bias due to labeling]: Thank you for highlighting this important nuance. You are correct that biases in human labeling can propagate to the classifier and potentially influence the results. However, by deploying our classifiers at high precision, we minimize domain contamination in the generated datasets, ensuring that our subsequent conclusions remain robust and valid. We have revised the sentence to: “Since human annotation for several million samples is expensive and prone to error, we rely on high-precision domain classifiers to filter the datasets.” This updated phrasing more accurately reflects our approach and its limitations. We have also added a paragraph regarding this to our extended discussion section (Sec. A.9).

[Q5: On the title]: Despite being a longstanding open problem, ever since the advent of web-scale datasets, we believe that the community at large has forgotten to channel efforts into principally studying domain generalization, as was done in the ImageNet/-R/-Sketch era. We, therefore, went with this title but are open to following the reviewers’ consensus and changing it.

We hope our detailed responses have thoroughly addressed your questions and clarified our contributions. We kindly request you to reevaluate our work in light of this clarification and the feedback from other reviewers.

[1] Radford et al, “Learning Transferable Visual Models From Natural Language Supervision”

[2] Abbasi et al, “Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models”

[3] Nguyen et al, “SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning”

[4] Fang et al, “Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)”

[5] Li et al, “Distilling Large Vision-Language Model with Out-of-Distribution Generalizability”

[6] Shu et al, “CLIPood: Generalizing CLIP to Out-of-Distributions”

[7] Ben-David et al, “A theory of learning from different domains”

Dear Reviewer r683,

Thank you for reading our paper and providing feedback. We address your concerns and questions below and aim to clarify our work’s key points:

[W1 and Q1 - Literature assumes that CLIP generalizes OOD]: Numerous studies implicitly or explicitly assume that CLIP generalizes OOD, as reflected in the following quotes. We now reflect this sentiment more clearly in the introduction (Lines 80–88).

• “zero-shot transfer is more an evaluation of CLIP’s robustness to distribution shift and domain generalization” [1]
• “several studies suggested that some Vision-Language Models (VLMs) such as the CLIPs, exhibit OoD generalization” [2]
• “VLP (Vision-Language Pretraining) models not only show good performance in-distribution (ID) but also generalize to out-of-distribution (OOD) data” [3]
• “models such as CLIP, ALIGN, and BASIC have recently demonstrated unprecedented robustness on a variety of natural distribution shifts” [4]
• “These models (CLIP, GLIP, etc.) have demonstrated enormous potential in, a wide range of downstream applications… especially for the out-of-distribution (OOD) samples” [5]
• and, finally, the paper you cited notes that “CLIP achieves OOD generalization in a task-agnostic way” [6].

[W1 and Q1 - Clarification of core insight]: As noted above, many works attribute CLIP’s high performance on OOD benchmarks to inherent robustness or emergent properties of large training datasets. However, our findings challenge this interpretation. While, as you point out, under the IID assumption, models are expected to perform poorly when an entire domain is excluded from training, this principle has often been overlooked when evaluating foundation models like CLIP. Our key contribution is to demonstrate conclusively that CLIP’s strong performance on these benchmarks stems from domain contamination in its training set, not intrinsic robustness or OOD generalization (see Sec. 3.3 and Sec. 4). To reiterate, when models trained on uncurated, web-scale datasets excel on fixed OOD benchmarks like ImageNet-Sketch, it’s often assumed they can generalize to arbitrary shifts [1–6]. However, if the training and test data are not truly distinct, these evaluations reflect in-domain performance rather than genuine OOD robustness. In Sec. 4, we demonstrate this flaw by ensuring the test data is genuinely OOD—removing similar/in-domain samples from the training set—and observing that CLIP fails to maintain its performance. This highlights that CLIP’s OOD performance on renditions critically depends on training data containing rendition images. Our work thus underscores a significant evaluation flaw, serving as a cautionary tale about the need for rigor in OOD evaluations and tempering expectations regarding CLIP’s true generalization capabilities. To better emphasize this core insight, we have revised the main text (Lines 80–88) and included a more detailed discussion (Sec. 6 first paragraph).

[W1.1 - Regarding CLIPood]: Our work explicitly examines the ‘zero-shot generalization’ capabilities of CLIP models. In this context, even CLIPood [6], observes that “vision-language pre-trained models demonstrate impressive zero-shot learning performance and outperform models trained from only labeled images, which reveals a promising approach toward OOD generalization” and that “CLIP achieves OOD generalization in a task-agnostic way.” [6] focuses on the degradation of this perceived generalization on certain tasks when adapting to another specific set of tasks, distinct from the problem we investigate in this work.

[W2 - Reliance on human-labeled data]: To our knowledge, there is no automated or principled method to reliably differentiate between natural and rendition domains. Moreover, as highlighted in Sec. 3.3, Tab. 9, and Appx. A.1.6, even widely-used single-domain evaluation datasets (e.g., DomainNet, ImageNet-Sketch) are contaminated with samples from other domains, making them unsuitable as training datasets for the domain classifier. Additionally, since LAION lacks domain labels and required domain classification for our study, we had to manually label data from the dataset to ensure the domain classifier’s accuracy and generalizability. That said, if clearly demarcated datasets—human-labeled or otherwise—exist for other domains, our methodology can be seamlessly applied to those settings, and we expect our conclusions to also hold there.

Dear Reviewer HGzr,
Thank you for your insightful feedback and for acknowledging the strengths of our work. We address your comments in detail below:

[W1 - Missing literature]: We appreciate you bringing these relevant works to our attention. We have incorporated them in the first paragraph of the Related Work section (Sec. 2) on Lines 132 and 133.

[W1 - Smaller training data size]: To clarify, the training dataset size of 19,000 refers to our domain classifier (Line 205 in Sec. 3.1). The CLIP models used in our primary experiments (Sec. 4) were trained on datasets ranging from 4M to 57M in size (see Sec. 3.4, Tab. 12, x-axis of Fig. 4). We now also include this information in Fig. 3 to make this point clearer. Additionally, we direct your attention to Lines 500–507 in Section 6, where we discuss the validity of our conclusions for larger dataset sizes. Based on your suggestion, we have further elaborated on our reasoning for the scalability of our conclusions in the second paragraph of Sec. 6:

The disparity in `relative corrected accuracy' shown in Fig. 4 remains stable across dataset sizes from 4M to 57M. Similarly, effective robustness illustrated in Fig. 5 is influenced by the training distribution rather than the dataset size, which is also supported by findings in previous works (Miller, 2021, Fang, 2022, Mayilvahanan, 2023). Lastly, CLIP’s performance scales predictably across domain mixtures as shown in Sec. 5. Overall, we see no indication that our results should not transfer to larger scales.

[Q1 - Is there a way to automatically curate the data splits?]: Yes! This is already part of our approach: We labeled 19,000 datapoints to train the domain classifier as described above, and used this classifier to identify 57M/16M natural/rendition images in LAION-200M. The same classifier could be used to label the domain of images from a larger pretraining distribution, if available.

[Q2 - Generate “Unnatural” Images]: This is an excellent idea, and was the motivation behind the ImageNet-C dataset. The specific corruptions employed in ImageNet-C are not OOD with respect to LAION, but other novel corruptions might be possible. We leave this analysis for future work.

Thanks again for your questions and feedback. We hope to have addressed all of your concerns.