CNS-Bench: Benchmarking Model Robustness Under Continuous Nuisance Shifts

I think the writing, especially in section 3.2 where the method is explained, could be improved quite a bit.

We rewrote that section and are open to any additional feedback about that part.

The legend of figure 7 is broken. [...] in figures 9 and 10 it might be better to share the y-axis for more comparability between the plots.

Thanks for pointing out that. We addressed the point in the updated manuscript.

Subscripts in line 197

We reworked the whole paragraph and we hope this addresses your comment.

In figure 3, how is it possible that the difference of two cosine similarities, which should be <= 2, achieves values of up to 7.5?

Thank you for raising this inconsistency. For this plot, we multiplied the cosine similarities by 100. We updated the plot accordingly.

In line 423, you write that an explanation for the abrupt change in failure rate of the cartoon style is the ImageNet class “comic book”, but I don’t see why images would be misclassified as comic books more for scale 1.5 than for scale 2 and higher.

First, we quantitatively evaluated our empirical observations and we report the ratio of classes that were wrongly classified as comic book for the cartoon shift:

These evaluations show that a significant part of wrong classifications (>50%) can be, indeed, attributed to the comic-book class. This also shows that the misclassifications increase: While the point of style change seems to be rather abrupt for a human, some visual properties continue to shift towards a more simplistic cartoon at later steps. Therefore, more and more images are misclassified.

Do you have any way of asserting that the severity levels of different shifts and different classes are actually calibrated? Since you are training different LoRAs for the different classes, I’m not sure if this will always be the case, but it might be desirable. (I guess one could calibrate this using the CLIP-distances…?)

We refer to our answer to your previously raised point about the calibration to address this comment (2nd point in this comment). Calibrating shifts is an exciting challenge for future work. Applying CLIP might be good way to achieve this. However, the CLIP measure is not always reliable for our structure-preserving shifts, as we exemplarily visualize in Fig. 16.

In principle, could you combine different distribution shifts at the same time? E.g., modify the same image to both exhibit fog and snow?

Yes, our LoRA adapters can be composed, as demonstrated in an example application in the supplementary material (Fig. 29). Exploring the robustness to combined nuisance shifts is indeed an interesting direction for future work.

Additionally, we would like to highlight a broader aspect of our method: it is not limited to nuisance shifts specified via text. Our approach has the potential to handle other types of nuisance shifts, such as a continuous distribution shift from ImageNet to ImageNetv2. However, for this work, we choose to focus specifically on more confined distribution shifts to maintain a clear scope.

One fundamental weakness of the paper is the lack of motivation for why a robustness evaluation at different levels is important.

We refer to the common response (i) where we discuss that considering multiple scales allows a more nuanced study of model robustness.

Why is it interesting at which severity level a model fails, especially given that it’s unclear whether the corruption severity levels across different shifts and different classes are properly calibrated against each other?

Calibrating shifts is difficult or subjective and maybe even impossible across different shifts, i.e., sand storm and painting, which are fundamentally different. Similarly, not all levels of different corruptions types in ImageNet-C can be directly compared. Therefore, our motivation for the failure point is grounded by the goal to check whether a model fails earlier or later than other models, which does not require a calibration across shifts but still enables us to already make important observations.

Of course, having a method of subjecting any training set to a natural distribution shift is great, but the Dataset Interface paper already achieves this.

We agree that the Dataset Interface provides a valuable basis for generative benchmarking. However, we believe that the contribution of being able to realizing continuous shifts is note-worthy since it allows a more nuanced and systematic study of robustness, showing that the ordering of models can change for different scales and shifts. Additionally, we enable more fine-granular changes by applying LoRA adapters. And lastly, we carefully analyze the effect of out-of-class samples and propose a more effective mechanism that better removes such OOC samples.

but I wonder why that matters, unless the ordering of models drastically changes across the different levels. I wonder whether the differences in figure 6b and 6c [...] are statistically stable.

To address your concern whether our findings are statistically stable, we plot the confidence intervals in Fig. 13 and 14, which shows that some of the depicted model rankings actually change significantly. Therefore, we underline that aggregating the performance into one single metric removes relevant insights when evaluating the OOD robustness in a specific scenario.

If I had a dataset with a painting-corruption, how would I know what the corruption-scale of my dataset is, to then select the best model at that level?

We agree that our work does not address this question. While we focus on benchmarking along our pre-defined scales, estimating the shift scale of a specific image or dataset is a fundamentally different task that goes beyond the scope of our work. However, our trained LoRA adapter might be potentially applicable for this task, which is an exciting future direction. The adapters scales could be estimated using the standard diffusion noise prediction objective, providing a measure of the average shift scale of a dataset or even a single image.

And do I really care about the minuscule differences between models (<< 1% accuracy delta) at scale 1, or would I simply select the model that does best at the maximum scale?

Ideally, one would choose a model that performs best for all scales. However, we, again, follow the same motivation as ImageNet-C (see common response (i)): If one chooses simply the model with the best performance for the largest scale, one might end up using a model that performs worse for all other scales. Let's consider a specific example for the sandstorm shift (as in Fig. 34b): A model that only focuses on the front perspective of the head of the dog, will outperform a model at the last scale even though it might perform worse for most other images of that class.

While I appreciate that the authors included the failure cases in figure 16, they do make me wonder how reliably the method really works [...] It would be good to also add confidence intervals to figure 3.

Fig. 16 (Fig. 26 in the updated manuscript) shows failure cases of the naive strategy for achieving continuous shifts (interpolation of text embeddings) and explains why we did not pursue this simple strategy and, instead, worked with LoRA adapters. An example failure case of our method is depicted in Fig. 2, which our filter strategy removes. As stated in the common response (iii), we conducted a human assessment that showed that 1% of our images are not samples of the class. We hope this also addresses your concerns. We plot the confidence intervals for Fig. 3 in Fig. 15.

I would have liked to take a closer look at the images in the benchmark, but could not unzip the provided benchmark.zip file, apparently because the file was corrupted. I don't think it's an issue on my end, could you look into this?

The benchmark.zip file works on our end, but Google may show warnings with large files. We’ve uploaded a subset of the dataset in CNS_dataset/example_imgs on Google Drive. We hope this helps!

Calibrating shifts is difficult [...] I agree, hence my question about this - I don’t think it is legitimate to calculate the failure point by averaging over the different shifts, since the different shifts could be on very different failure scales.

We agree with the concern of the reviewer and removed results with averaged failure points (Tab. 4 in the previous manuscript). Please note that we motivated the failure point computation for individual shifts (l. 465-472).

I appreciate this, and find the confidence intervals smaller than expected (I assume this is a 95% confidence interval?)

The plots in Fig. 13 and 14 depict the one-sigma confidence interval, as mentioned in the caption. To address your original comment whether model rankings actually change significantly, we perform a statistical test: We test whether the estimated accuracy drops are significantly different using the two proportion z-test and a p-level of 0.05. For example, for the painting style nuisance we find that the accuracy drop differences between RN152 and ViT, ConvNext and ViT, DeiT and ViT, DeiT3 and ViT significantly change the sign when increasing the severity of the shift. We further checked which shifts result in significant model ranking changes for the considered models along the architecture axis. We observe such statistically significant changes for the following shifts: painting style, the style of a tattoo, heavy sandstorm.

Currently, section 3.1 still is not self-containing, as somebody unfamiliar with the work of Vendrow et al 2023 and Gal et al 2023 will have a hard time understanding what was done. It would not hurt to explain a bit more of what was done here. In line 194 there’s a “that” which doesn’t belong there.

Following your initial feedback, we specifically addressed the criticism about the missing clarity of Sec. 3.2. Now, we also updated section Sec. 3.1. We hope this revised version is clearer and we are looking for your feedback. We are willing to add a more formal explanation of the diffusion model objective if you think, this should not be missing.

Please let us know if you see potential for more clarity in other parts of the paper. We are happy to address these points as well. Thank you for your thorough and constructive feedback!

That’s interesting, thanks. What I was wondering about was why the FP ratio would decrease for the larger scales again (I would have expected the results to look like your table does, with more mis-classifications at the larger scales).

The failure point distribution quantifies when a classifier starts failing (per scale), which rather relates to the performance drop with respect to the previous scale. In contrast, the presented table measures the absolute accuracy drop per scale with respect to scale 0. In that case, the drop from scale 1 to scale 1.5 is the largest (0.13-0.02=0.11). That means, the classifier starts failing more often at intermediate scales for such shifts.

Thank you again for your response. We will address your comments individually:

[...] the accuracy drop you refer to in e.g. figure 5 is the absolute delta in classification performance, not the relative one. In other words, the accuracy of a ResNet-50, which should sit at a performance of about 80%, drops by 0.6 percentage points (e.g. to an accuracy of 79.4%) on the largest scale of your dataset, is that correct? Are you seriously proposing an "OOD-benchmark" for which even the hardest corruptions reduce the performance of a vanilla ResNet-50 by less than 1%? Please clarify this point.

We are sorry for the misunderstanding and we would like to clarify the scale of the y-axis: The y-axis is, indeed, the absolute performance drop. However, we do not report in % but in the value range [0,1] throughout our whole analysis. Therefore, the correct interpretation of the graph is that, for example, ResNet drops by 60 percentage points for the nuisance shift 'cartoon' (i.e. 0.6). We updated the caption of the figure accordingly and also mentioned it explicitly in the experimental details (Metrics) to ensure a proper understanding.

The “dust”, “rain”, “sandstorm”, “fog”, and “snow” categories do in my opinion not pose an OOD scenario at all, since they do not even occlude the object, but merely change the background slightly.

Occlusions of objects are indeed an interesting OOD factor to consider. However, OOD scenarios can also include many other factors of distribution shifts, e.g., texture variations, high frequency corruptions or background changes. While the applied nuisance shifts in terms of weather conditions are sometimes not as drastic as one might expect, please note that their application already significantly reduces the performance of the classifiers (e.g., 20% for ResNet-50 for snow) and, hence, clearly induces an out-of-distribution shift of the data. Please also consider that the presented subset includes images with varying scales and some images actually include (partial) occlusions.

This came as a surprise to me, because I don't see how these images would be harder for any reasonable model than uncorrupted images.

It is a known phenomenon that classifiers make mistakes that are rather unexpected for humans. For example, it was shown that classifiers can fail to recognize objects due to simple changes that would not fool a human observers, such as changes in the object pose [A], the background [B], or the texture of the object [C]. This observation is commonly attributed to the "shortcut learning" phenomenon [D], where models rely on simple, often misleading patterns in the data, rather than understanding the underlying concepts. In this context, our framework is the first that enables the systematic evaluation with respect to diverse and continuous real-world nuisance shifts, providing deeper insights into model robustness.

[A] Michael A. Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. "Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[B] Rosenfeld, Amir, Richard Zemel, and John K. Tsotsos. "The elephant in the room." arXiv preprint arXiv:1808.03305. 2018.

[C] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." International Conference on Learning Representations. 2018.

[D] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. "Shortcut learning in deep neural networks." Nature Machine Intelligence 2.11: 665-673. 2020.

Dear reviewer 1iK7,

thank you for your response and your active engagement in our discussion. This clearly helped us improve our paper. We are grateful that you recognize the significance of our work and you agree with the other reviewers that our work represents a valuable contribution to the community.

The Authors

Why were those 14 "nuisances" chosen and not others, why 14 and not, say, 50?

We selected those diverse shifts as an example application of our approach, mainly inspired by ImageNet-R (8 shifts) and real-world weather shifts (6 shifts). Due to the scalable nature of our generative benchmark, our framework can be used for computing the robustness with respect to other shifts that can be expressed via a text prompt or a set of images as well. We are eager to test others shifts that the reviewers might have in mind.

What's the robustness of a failure point to random variation?

We are not entirely sure if we understood this question correctly and are happy to clarify further if we did not address this point accurately: Which random variation does the reviewer refer to?

Different generator seeds lead to different generated images whereas for a given seed and scale the resulting image generation is deterministic. Therefore, we assume the reviewer refers to situations where a model classifies wrongly for an earlier scale but correctly for a later scale. The failure point metric is based on the first failing scale, comparing models on individual starting images to determine when the model begins to fail. If a model correctly classified a larger scale after having failed for a smaller scale, the cumulative number of failure points would be higher than the accuracy drop.

Is performance (accuracy) always a monotonous function of the LoRA slider strength? Are there instances when that's not the case? If so, what does it mean if there are images beyond the failure point that are again correctly recognized?

While the effect of the LoRA adapter is a monotonous function, it can effect the evaluated classifiers differently. Therefore, we observe few cases where a model fails first for lower scales and classifies correctly again for larger scales. Here, the failure point metrics differs from the average accuracy drop in a way that it is more sensitive to slight variations: We purposefully define this in a way that the first failure case is considered. The failure point means that there was no mis-classification before that scale.

line 43: "such approaches are not scalable" - why not? If one takes a large dataset and applies cheap corruptions like the ones from ImageNet-C, should that be considered less scaleable?

ImageNet-C corruptions are very simple and fundamentally different to real-world distribution shifts. Our approach is different to the proposed strategy since it can achieve diverse and more realistic distribution shifts without relying on manually annotated datasets, such as (Zhao et al., 2022). Therefore, scalability not only refers to the number of classes and samples but in particular to the number and complexity of distributions shifts that can be applied.

What's the computational cost of generating the dataset?

Training the LoRA adapters took 2000 GPU hours and generating the images took around 350 GPU hours. We also refer to Sec. A.5.6.

Figure 7: [...] I'd recommend using different colors here to avoid confusion.

Thanks for that note. We updated the figure accordingly.

Nuisance shifts affect information that's not related to the nuisance concept. [...] As a consequence, failures cannot be attributed to the nuisance concept itself.

Our strategy allows applying shifts that were not possible previously. However, using a generative model for realizing shifts inherently comes along with confounders. This is explained by the biases in the training data of the generative model. So, similarly, real-world OOD datasets also exhibit such confounders and it is equally hard to differentiate whether an accuracy drop can be, e.g., explained by

• a style change or a change of shape as in the ImageNet-R dataset (consider Fig. 24 for examples) or
• the heavy snow or the lower visual quality in the OOD-CV dataset (as already depicted in Fig. 31).

In contrast to previous works, we particularly reduce the confounders of the classified object by applying LoRA adapters only to later noise steps throughout the diffusion process, which prevents significant changes of spatial structure of the image. Therefore, while future benchmarks similar to our proposed one could benefit from better controllability and removed confounders to acchieve "pure" shifts thanks to the continued progress in generative models, we argue that our approach still achieves valid and confined distribution shifts that capture the real world biases.

The approach is based on generative models, thereby introducing a real vs. synthetic distribution shift that may further influence results. A discussion - better yet: an analysis - of this likely confound is recommended.

We agree that this needs to be considered carefully and fundamentally challenges any generative benchmark. To reduce the bias of the accuracy estimate between the distribution of Stable Diffusion and ImageNet, we used the ImageNet-trained text embeddings to better replicate the ImageNet distribution, which reduces the bias of the accuracy estimate by around 7%. We underline that generative benchmarks do need to address the biases arising from the real vs. synthetic shift. We use a filtering mechanism that is parameterized on a human labeled-dataset to reduce the biases of the accuracy estimates, we performed a user study to check the realism of the benchmarking images, and we compared the distribution shifts of a real-world dataset (OOD-CV) to our work. We discuss our strategies to address the realism of the generated images in the common response (iii).

Does this address your comment or do you propose a different way to analyze confounder?

The paper's main claim to fame remains a bit unclear to me. [...] I recommend that the authors clearly state and justify what they believe is the primary novel contribution ("claim to fame")

We really appreciate this very valuable comment and for re-framing the variety of options. We present our paper's claim to fame, i.e. our main contribution, as enabling the testing of the robustness of vision models with respect to realistic continuous-scale distribution shifts. We refer to the common response (ii) for a more elaborate discussion of the claim of fame.

We additionally address your other proposals: While we do not focus on improving the controllability of the generative model, we aim at selecting the best methods for achieving controllability in diffusion models. In general, we would like to support that generative benchmarking can be a complementary way for benchmarking models - not reducing the importance of real-world datasets. The advantage of generative benchmarks lies in the flexibility and scalability of possible shifts, classes, and samples. Additionally, sampling from a generative models better captures the statistics of the real world than a small dataset since it is trained on very lage-scale datasets. Reducing the biases of generative benchmarks through the removal of failure cases of generative models is necessary to advance in the realm of generative benchmarking. Therefore, we propose an improved filtering mechanism in our work and show that failure cases are effectively filtered out.

If it should be a benchmark, people will want to know: who won? What's the score that I need to beat in order to be SOTA?

Thank you for raising this point. We added Tab. 1 in the main paper and Tab. 2 in the supplementary where we present the mean relative corruption error as introduced for ImageNet-C. This metric allows ranking various models using one single quantity. However, underlining our key finding, we motivate that benchmarking robustness should not only involve averaged quantities.

Thank you very much for your response. We agree that this point is, indeed, important to consider for people using our benchmark or researchers working on improving generative benchmarks.

We added the following at the end of the method sliders section (l.232 in the updated manuscript): "Since our sliders do not explicitly exclude confounding variables, the applied shifts may also affect confounders inherently present due to biases in the training data. For example, as shown in \cref{fig:shift_dust}, using the in dust slider unintentionally removes half of the people, and in \cref{fig:shift_video_game}, the background no longer represents a forest. Consequently, failures in our subsequent analysis cannot always be solely attributed to the nuisance concept itself."

We also added a section about the limitations in the conclusions (l.531): "While our approach allows for diverse continuous nuisance shifts, it does not eliminate all confounders, meaning failures cannot always be solely attributed to the targeted nuisance concept. This highlights an inherent challenge for generative benchmarking approaches, and future advances in generative models could help mitigate these confounding factors. Additionally, while we have carefully addressed this issue in our work, we acknowledge that using generated images can lead to biases arising from the real vs. synthetic distribution shift.

We hope this benchmark can encourage the community to continue working on more high-quality generative benchmarks and to adopt generated images as an additional source for systematically evaluating the robustness of vision models."

Please let us know if you have other concerns.

Thank you very much again for your valuable review.

Dear reviewer zdn7,

we appreciate your positive rating of our work. Thank you again for your very constructive feedback, which helped us sharpening our listed contributions.

The Authors

I don’t understand the failure point concept in Section 3.2.

We revised that section and we removed the indices for a more simplistic notation. We are happy to get your feedback on the updated version. In essence, the failure point distribution captures at what shift scale a classifiers fails how often. Mathematically, we define the failure point distribution through a histogram that counts the number of failure cases that occur at one of the shift scales that our benchmark considers.

Could you provide some theoretical or empirical evidence to justify the rationale for adjusting LoRA for the last 75% of the noise steps?

Our goal was to maintain the coarse semantic structure of the image. Therefore, we do not activate at earlier timesteps. We mainly guided our parameter choice following Gandikota et al. (2023). Consider Fig. 13 in their supplementary for this, where they show that 75% results in a significant application of the slider concept with spatial structure preservation. Our initial experiments confirmed this. Varying this value to analyze different nuisance shifts can be an interesting step in the future.

Was the data used for fine-tuning the entire CNS-Bench? [...] after fine-tuning, the model accuracy on IN/val decreased by 2.04%.

We used the entire CNS-Bench for fine-tuning. The reduced performance could be attributed to the fact that the model has to stop considering details it has learned for differentiating ImageNet classes, to improve the ImageNet-R performance. However, stylization requires less focus on the texture, which can eventually deteriorate the ImageNet performance. We refer to the common response (iii) for a discussion of the realism of the generated images.

please further elaborate on the rationale for using the slope of the linear fit between ID and OOD accuracies and the significance represented by this slope. Why not use the linear correlation coefficient?

Our analysis of the relation between ID and OOD accuracy was mainly guided by the accuracy-on-the-line discussions (Miller, 2021), who find that a linear fit often captures the dependency between a ID and OOD accuracy surprisingly well. The linear slope is motivated since it quantifies which improvement of the OOD accuracy can be achieved on average when improving the ID accuracy. It helps better understanding whether an increase in OOD accuracy can be explained by a larger ID accuracy. Fig. 17 shows that most linear fits are statistically significant. To address your comment, we additionally computed the linear correlation coefficient and plot it in Fig. 18.

Furthermore, please provide a more detailed analysis of the results in Figure 7.

The results in Fig. 7 show that the slope varies for different (1) shifts and (2) scales. (1) This is in line with the discussions by Miller et al. (2021): The slope varies for different datasets, i.e. the effect of a better OOD accuracy with an improved ID accuracy is not consistent across shifts. (2) The smaller slope for lower scales can be attributed to the subset of classes that we consider in our benchmark: A delta accuracy increase on all ImageNet classes leads to a smaller delta accuracy increase for our selected generated subset. We illustrate two examples of such a linear fit in Fig. 19. However, when the distribution shift is more prominent, an improved ID accuracy leads to a more prominent improved performance of the OOD accuracy. Our strategy allows systematically studying the effect of different distributions shifts for the relation between ID and OOD accuracy.

Figures 6a and 6b evaluate the accuracy drop. I do not think this metric rational because the model size and performance on the ImageNet validation set may not necessarily align. This mismatch could result in accuracy drops of different models that are not directly comparable. Please provide the model's parameter count and the model accuracy on IN/val for reference or other evidence to claim rationality the accuracy drop.

Our benchmark mainly focuses on the accuracy drop to measure the relative OOD robustness of a model or the performance degradation for the case of nuisance shifts, following Hendrycks et al. (2018,2021). But we agree that there exists a relation between ID and OOD accuracy. We considered and reported the relation between ID and OOD accuracy in Fig. 7. To additionally address your comment, we report the ImageNet validation accuracies and the model parameter counts in Tab. 5. While the model parameter count purposefully varies for the model size axis, we use the same ViT-B backbone for the pre-training axis, and a comparable number of parameters for the architecture axis. To further address your question, we additionally compute the accuracy gain after subtracting the effect of an improved OOD accuracy taking into account the linear fit that we discussed previously. We plot the results along the model architecture axis in Fig. 19. We find that ConvNext achieves the best accuracy gain after removing the linear dependency of an improved ID accuracy. However, we underline that this measure needs to be considered cautiously since the (1) linear fit depends on the selected models for computing the statistics and (2) the linear fit might not always describe the relationship sufficiently well. Consider, e.g., the discussions in 'Accuracy on the Curve: On the Nonlinear Correlation of ML Performance Between Data Subpopulations' by W. Liang (2023) or in 'Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation' by A. Sanyal (2024). Therefore, we would rather argue for consistently using the accuracy drop as a measure of robustness. We are happy to get your feedback on this point.

Figures 4 and 5 assess using accuracy, while Figure 6 employs accuracy drop. Could you standardize to a single metric for consistency throughout the text?

Thanks for pointing out that inconsistency. We updated the figures accordingly.

What are the distinctions between CNS-Bench and ImageNet-C?

Imagenet-C does not contain real-world shifts but only simple corruptions, which is a fundamentally different nuisance shift, not capturing all realistic distributions shifts. We also refer to the discussion in the general comment (i).

Could you give some experiment details of the claim “the alignment for one given seed increases in 73% for scales s > 0 for all shifts in our benchmark” in Section 3.2?

To evaluate whether the application of our sliders yields an increase of the desired shift, we compute the CLIP alignment of the generated image to the text prompt describing the shift. Increasing the scale $s$ by 0.5 increases the CLIP alignment in 73% of the cases. This shows that the shift is increased in the majority of the cases when relying on the CLIP alignment score. An explanation for cases where the CLIP score does not increase can be that the applied shift by the LoRA slider does not exhibit the characteristics that CLIP measures for that shift although the change can be visible for a human. We illustrate such a case in Fig. 16, where the painting shift can be observed but the CLIP alignment to the shift drops. We moved this part to Sec. A.5.4 in the supplementary and added explanations there.

Dear reviewer LxGf,

thank you for recognizing the significance of our contributions and updating your score. Thank you again for your constructive feedback, which helped improve the clarity of our paper.

The Authors

Unclear contributions: The contributions listed in the paper seem overlapping.

Thank you for pointing out this. We re-structured our contributions differently to reduce the overlap and we refer to the common response (ii). The new structure is that we separate (1) the framework for continuous shifts, (2) the OOC filtering strategy, and (3) the evaluation and analysis of various models.

While the paper claims 14 distinct nuisance shifts as a key contribution, it lacks an explanation or rationale for selecting these specific shifts.

We added the motivation for the 14 shifts in the introduction and in the experimental details. In short, we selected those diverse shifts as an example application of our approach, mainly inspired by ImageNet-R (8 shifts) and real-world weather shifts (6 shifts). Due to the scalable nature of our generative benchmark, our framework can be used for computing the robustness with respect to other shifts that can be expressed via a text prompt or a set of images as well. We are eager to test others shifts that the reviewers might have in mind.

Ambiguity in benchmark superiority : The authors assert that their benchmark outperforms existing benchmarks for evaluating model robustness by incorporating nuisance shifts across multiple severity levels. However, earlier works by Hendrycks & Dietterich (2018) and Kar et al. (2022) already support multi-severity analysis for vision model failure points. Thus, the authors should clarify how their benchmark framework distinctly advances beyond these existing approaches.

While the mentioned works allow multiple severity levels, they are restricted to synthetic or simple semantic corruptions. Our approach allows real-world distribution shifts and can be easily scaled to other shifts. Applying diverse and realistic natural real-world shifts is fundamentally more challenging and motivates the application of generative models. We do not consider our work as superior but complementary to the benchmarks that support simple multi-scale corruptions.

We also refer to our common response (i).

Inconsistent statements on model robustness: In line 451, the authors claim that transformers are more robust than CNNs, yet this statement seems contradicted by Fig. 6a, where ConvNext outperforms ViT and DeiT but performs slightly worse than DeiT3. This inconsistency suggests that CNNs may not always be less robust than transformers, and the statement should be re-evaluated or clarified.

We believe there may be a misunderstanding. Our findings indicate that CNNs generally perform worse than transformers when using modern training recipes like DeiT3, which we also tried to communicate. We noted in the paper: "A modern CNN (ConvNext) outperforms transformers but is less robust than when using modern training recipes (DeiT3)." (l.451 in the original version). We apologize for the confusion and we reformulated that paragraph for clarity.

Validation of realistic nuisance shifts: While the authors argue that the benchmark includes realistic nuisance shifts, the realism of these diffusion-generated images is not substantiated. Proper validation, such as human assessment, would enhance the credibility of this claim.

Thank you for raising that important point. We refer to the common response (iii) for a discussion of our strategies to account for the realism. Specifically, following your advice, we also validated the realism of our images through human assessment, which we document in Sec. A.7. One percent of our benchmarking samples were detected to be not a sample of the desired class according to our user study where each image was checked by two different individuals.

Readability of figures: The font size in several figures is too small, which detracts from readability. Increasing the font size would improve clarity for readers.

Thank you for that comment. We updated the figures accordingly.

Self-supervised pre-training: Why is DINOv1 using linear probing compared with other models? This seems to create an unfair comparison, as linear probing may not fully reflect the robustness of self-supervised models relative to other models in the evaluation. Could you clarify the rationale behind this comparison approach?

Originally, we only compared using the publicly available linear-probed DINOv1 model since no fine-tuned model was available. To address your comment, we fine-tuned DINOv1 and updated the results and discussions accordingly (consider, e.g., Tab. 1,2,3 and Fig. 6). The fine-tuned DINOv1 model achieves a comparable performance to MoCov3 on CNS-Bench, with a better performance on small scales but achieving slightly worse results on larger shift scales.

Dear reviewer xh13,

thank you for recognizing the significance of our contributions and updating your score. Thank you again for your critical and constructive feedback, which helped us more clearly state the relevance of our benchmark.

The Authors

The novelty of the insights presented in this paper could be more compelling. For example, in Figure 6, are there any underlying reasons or mechanisms that could provide a deeper understanding of the results?

Thank your for pointing out that our discussions were not clearly stating the novelty of our observations. We try to formulate the key insight of our benchmark in the following more clearly.

First of all, we find that there is not a single model that rules all realistic shifts and scales equally well. When averaging the performance over all shifts, model rankings do not heavily change for different scales (Fig. 6a). This states that a robust model that is robust on a weakly-shifted OOD dataset A tends to be robust as well on heavily-shifted dataset B. However, considering this average metric is not sufficient to evaluate the robustness of a model on a specific OOD scenario: When comparing the performances for individual nuisance shifts, the model rankings can significantly change for different scales. This effect depends on the considered shift and models. E.g., we observe that the effect of weather variations, such as rain or fog, results in varying performance for different scales and shifts but increasing the scale does not significantly change the model rankings (Fig. 9). This is similar to ImageNet-C, where different corruptions lead to different performance drops over varying scales (Fig. 22). However, some style changes impact the models clearly differently for various scales (Fig. 6b and 9). We believe that this might be attributed to the effect that different models focus on different characteristics of the classes, which are modulated differently at different scales of the shift, which, we think, is a note-worthy novel finding.

We hope, this addressed your comment appropriately. Depending on your feedback, we will update the discussions in the main paper accordingly.

There is a lack of clarity on how the dataset handles potentially unrealistic or counter-intuitive scenarios, such as cars driving on water.

We believe there are two points to address this question. Our benchmark follows the statistics of the training data of the generative model, which captures the distribution of available images. Unrealistic cases can however still happen but it is typically hard to automatically differentiate between edge cases and physically implausible or unrealistic cases. Consider, e.g., the presented examples in Fig. 30 that relate to the example of cars driving on water.

Nevertheless, it could be argued whether such unrealistic or counter-intuitive scenarios are problematic as long as the class is still recognizable since humans also generalize to edge cases. It might be even a motivation to use generative models for benchmarking to generate edge cases that only rarely occur in real world scenarios. Thank you for raising this point.

We hope our answer addressed your question appropriately. Please feel free to follow up if you have any remaining questions or concerns.

Dear reviewer 8WvD,

we hope, our rebuttal addressed your concerns. Thank you again for your positive review and for recognizing the significance of our work.

The Authors