LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models

Dear Reviewer,

Thank you for your favourable review. We appreciate that you find the utilization of LAION-C logical because it poses a new challenge for OOD-evaluation, and value the human experiment. Prompted by your feedback, we will update the manuscript to include a section motivating our artificial corruptions better, and publish a version of our benchmark that consists of 1,000 class labels (LAION-C-1K). Thank you for alerting us to the existing nomenclature regarding the different types of distribution shifts (semantic vs covariate shifts); we have updated the manuscript accordingly. Our shift is indeed of the covariate shift kind.

Q: “Did you take into account that the ImageNet has .png (validation set) images and the Laion-C has .jpg compressed images?”

A: Thank you for raising this point. We double-checked that ImageNet images are JPEG-compressed in both training and validation sets. Since LAION-C base images were taken from the ImageNet validation set, analogous to ImageNet-C, this design choice matches other benchmarks. We therefore also use JPEG-compression, setting parameters which give close to lossless compression.

Q: “Are 19 evaluators a good number?”

A: Valid question. Yes, 19 evaluators are a sufficient and typical number for this type of study: Since each human observer is exposed to 2 corruptions, every corruption is seen by at least 6 different humans, assuring good coverage. For reference, [1] used only 4 humans per corruption for most of their corruptions. In psychophysics, this type of design is called a “small N design” (fewer participants that see many images, [3]). More quantitatively, we see in Fig. 10 that the 95% confidence intervals surrounding our estimate of the average human performance are quite small (+- 4.61%). Statistically speaking, it is therefore very unlikely that adding more human observers would change any of the results or interpretations. We will add a detailed discussion of this to the Appendix. Since beating the best human is more difficult than beating the average human, we analyze peak performance rather than average performance in Fig. 5 and it is impressive to see that good models beat every single human. The confidence interval is smaller for the models because they were evaluated on all images, while each human only saw a subset of images for practical reasons. In the Appendix, we will now include a comparison to both average and peak human performance."

Q: “Why only 16 classes?”

A: We used 16 classes to enable a comparison to human observers in line with earlier work [1, 2], because humans cannot be expected to reliably classify images into too many categories, simply due to pragmatic limitations (size of the response icons, time per trial etc.) Prompted by your suggestions, we have decided to also provide a version of our dataset consisting of all 1,000 ImageNet classes, which we call LAION-C-1k. The dataset is ready and has been submitted for review on Zenodo. While we are not including a direct link at this stage respecting the rebuttal policies, we will link the dataset in the camera-ready version.

Q: “How should I use the different intensity levels?”

A: We included different intensity levels so that practitioners would be able to see the decline in model performance and obtain a more fine-grained evaluation of robustness. This is done by some other OOD datasets as well, like model-vs-human [1] and ImageNet-C. The protocol is to evaluate a new model on all levels of our benchmark and report the average across levels for the benchmark result (we’ll make sure to make this clear in the manuscript and github to ensure comparability). Additionally, users are free to report performance curves like we do in e.g. Fig. 5.

[1]: Partial success in closing the gap between human and machine vision (Geirhos et al., 2021)

[2]: Generalisation in humans and deep neural networks (Geirhos et al., 2018)

[3] Small is beautiful: In defense of the small-N design (Smith and Little, 2018)

Dear Reviewer,

thank you for taking the time to provide such detailed and insightful feedback. We were delighted to read that you found our experimental design [to be] in general thoughtful and comprehensive, and appreciated the substantial evaluation and insightful error analysis. We have clarified the manuscript based on your suggestions, added citations, and scaled the dataset to a substantially larger version with 1k classes, for a total of 1.5M images.

Q: Motivation for choosing these 6 types of distortions.

Our distortions needed to fulfill the desiderata of 1. being exotic enough to have a low occurrence probability even in web-scale datasets and 2. testing relevant feature extraction capabilities of the models. We now include this motivation in Sec 2.1. To solve the Stickers and Mosaic distortion, a model needs to be able to holistically integrate the image, instead of being led astray by local image cues induced by sub-images, which is notoriously difficult for DNNs [1]. The Glitch and Vertical Lines distortions are the most exotic and globally disruptive image transformations we could find, and destroy the texture cues that models rely on [1]. The Geometric Shapes distortion tests amodal completion, a staple of human visual processing even in infants [2,3]. They also change the color distribution of the image, which humans are robust to because we do not rely primarily on color for object recognition [4,5]. The Luminance Checkerboard distortion tests a model’s ability to adapt to local lighting conditions, an important skill of the human visual system [6].

[1] Geirhos et al 2019

[2] Kellmann and Spelke 1983

[3] Nanay 2018

[4] Tanaka and Presnell 1999

[5] Biedermann and Ju 1988

[6] Heeger 1992

Q: not much discussion or contrast with other OOD dataset

Thanks for the suggestion - we have added a figure contrasting LAION-C with other OOD datasets: https://ibb.co/HfZmLkTd. This shows that LAION-C offers better resolution of model differences. Furthermore, in the camera-ready version we will substantially expand our discussion of other OOD datasets.

Q: I don't think the Table 1 result is sufficient.

Thanks for the great suggestion. While our initial approach demonstrates learnable signals in LAION-C, your suggestion offers a stronger proof of concept. We now fine-tuned the same model backbone as in Tab.1 on a mixture of clean and distorted ImageNet-Train images and evaluated on both ImageNet-Val 16 class and LAION-C 16 class. The results illustrated here (https://ibb.co/rGmQk2rG) indeed show that ‘improvement of LAION-C accuracy can be achieved without regression on the clean dataset’. We will update Sec. 3.3 accordingly with the results and full experiment setup to strengthen our argument.

Q: LAION-C only has 273 images

We selected 273 images per class, yielding 4,368 base images and ~130k total images after corruption at different intensity levels. Nonetheless, we hear your point regarding broad coverage and scale. To address it, we extended our dataset to the full 50k IN-validation set, resulting in LAION-C-1k with 1.5 million images. The dataset is ready and has been submitted for review on Zenodo. While we are not including a direct link at this stage respecting the rebuttal policies, we will link the dataset in the camera-ready version.

Q: Why is it called LAION-C?

Great question. We follow common practice in the OOD literature, where datasets are named w.r.t. the dataset for which they’re OOD. E.g., ImageNet-Sketch is designed to be OOD for ImageNet (even though none of the sketch images are part of ImageNet since that would defeat the purpose of being OOD). We now explain this motivation in the introduction. Applying corruption to LAION images directly is possible, but they don’t have class labels, thus cannot be used (without proxy/pseudo labels) for classification.

Q: [M]ore justification on the human performance is needed.

Please see our response to ZcBa.

Q: LAION-C difficulty vs other protocols?

Our main results (Fig. 4) follow a cross-dataset evaluation protocol: The models were trained on web-scale datasets, fine-tuned on IN-1k but evaluated on LAION-C. In general, it is possible to apply any OOD evaluation protocol to LAION-C that one could also apply to other OOD-datasets, such as ImageNet-[A, R, C, Sketch].

Q: Why are sticker and mosaic distortions more challenging?

Great question. We introduced content-rich image tiles into the original image, to test the model's ability to process images holistically in spite of local distractions. These two distortions are built on simpler augmentations that mask or pixelate content, as illustrated in https://ibb.co/CpBvR2Y3. We observe an up to 35% drop on ViT-B32 compared to these easier distortions (https://ibb.co/hJj8pfQw), indicating that local signals pose a significant challenge. Fig.12 also supports this, as LLM-based models frequently direct attention toward the inserted tiles.

Dear Reviewer,

thank you for your insightful comments! We are delighted that you find our work to have a good experimental design and consider it relevant to the community.

Q: “More ImageNet OOD variants could have been studied / compared with”

A: Thanks for this excellent suggestion. We have added a comparison of LAION-C and several well-established OOD datasets to the paper, see https://ibb.co/HfZmLkTd, which shows that LAION-C captures the variance in model performance better than other datasets, with a standard deviation of ~27%, whereas other common OOD datasets, on average, have only ~10%. LAION-C is tested on a 16-class basis, while other datasets typically use 200-1000 classes, making this result even more remarkable. We want to note, however, that the point of LAION-C is precisely not to capture “naturally occurring” perturbations (see next question). We also added the missing reference to ImageNet-Sketch you pointed out (note the related work section in Appendix A.1, where we already refer to ImageNet-A).

Q: “[T]he synthetic nature of the perturbations questions whether they represent real world occurrences [...]”

A: Indeed, LAION-C is designed to be synthetic, just like ImageNet-C for ImageNet. “Natural” images as used by other common OOD datasets are scraped from the web, making them no longer OOD in an era where large multimodal foundation models are trained on nearly the entire web, for multiple epochs: as of 2025, we need to assume that most natural images found on the web were part of the training corpus. Robust representations, however, should generalize to challenging images (e.g. identity-preserving transformations) no matter the type, both synthetic and natural—just like human perception is robust to many different types of OOD data. Being able to handle challenging synthetic input typically transfers to unexpected natural input, as evidenced by the “sim2real” line of work e.g. in robotics and autonomous driving.

Q: “Have the authors considered constructing a OOD test set in a similar manner to the ImageNet-A dataset but for LAION to test real world failure cases?”

A: Thank you for the suggestion. In the current era of large-scale training, we have to assume that models will eventually be trained on the entire internet, hence scraping the web for benchmark images, as was done for ImageNet-A, is no longer a feasible choice for obtaining an OOD dataset.

Q: “The paper oversells the paradigm shift claim as the perturbations proposed are in the same vein as the ImageNet-C corruptions.”

A: This might be a misunderstanding. The “paradigm shift” we refer to is the shift from humans outperforming models (as observed in prior work [1,2,3,4,5]) whereas now, based on our carefully collected human comparison data for LAION-C, the best models outperform humans. Apologies if that wasn’t clear, we’ll make sure to highlight and explain this better.

[1] Partial success in closing the gap between human and machine vision (Geirhos et al., 2021)

[2] Comparing deep neural networks against humans: object recognition when the signal gets weaker (Geirhos et al., 2017)

[3] ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpoints (Dong et al., 2023)

[4] Recognition in Terra Incognita (Beery et al., 2018)

[5] Why do deep convolutional networks generalize so poorly to small image transformations? (Azulay & Weiss, 2019)