Assessing Visually-Continuous Corruption Robustness of Neural Networks Relative to Human Performance

We appreciate the reviewer's comments on the difficulty to understand the text. We have revised Section 2 (Methods) as suggested, and replaced "toolbox" by "code".

Most importantly, we have also created a visual summary of the VCR metrics in Section 8.4, and referred to the figures from Section 2.

Further, we have added a more precise, but still high-level description of how VIF is computed, using wavelet decomposition and mutual information, in Section 8.5. The detailed formulas could easily fill an entire page but are available in the original paper [Sheikh & Bovik, 2006].

Finally, regarding impulse noise or glass blur: impulse noise is a color analogue of salt-and-pepper noise, and glass blur simulates frosted glass windows or panels. Both corruptions are included in ImageNet-C. All of the corruptions in our study are visualized in Figure 10 in Section 8.6 of the appendix.

We very much appreciate the valuable comments and questions. We address each of them individually.

"In this paper, The VIF is adopted as the quality measure. However, compared with the advanced full-reference quality measures, such as LPIPS [1], and DISTS [2], the VIF is usually inferior."

We choose VIF, since it is well-established, computationally efficient, applicable to our transformations, and still performing competitively compared to newer metrics. In fact, according to Table 1 in DISTS [Ding et al., 2022], VIF performs better than LPIPS across all three datasets used in the comparison, and it outperforms DISTS on one of them. Even though DNN-based metrics like DISTS may be applicable to a wider class of transformations than VIF, including those that affect both structure and textures, their scope may depend on the training datasets in potentially unpredictable ways. On the other hand, the scope of VIF is well-defined based on the metric's mathematical definition. In particular, VIF is suitable for evaluating corruption functions that can be locally described as a combination of signal attenuation and additive Gaussian noise in the sub-bands of the wavelet domain. The transformations in our experiments are local corruptions that are well within this scope. We have updated Section 8.5 to reflect this rationale. However, future work should explore VCR using other IQA metrics.

"In Sec.3, Page 4, the coverage between the IMAGENET-C and VCR Test Set is compared by splitting the full quality range into 40 bins. However, the coverage is highly relevant to the number of bins. In an extreme case, when the bin number is 1, the same coverage the two sets will possess. As such, how to ensure the coverage is reasonable?"

We follow the established standard for the number of recall positions when calculating average precision, which are also empirical performance curves with a similar shape. For example, the KITTI benchmark has increased this number to 40 from 11, which was originally used in the PASCAL VOC benchmark.

"During subjective testing, human decisions are usually highly affected by the memory effect, i.e., a severely corrupted image could still be recognized successfully when humans have observed the same image content but with a high quality. The authors should illustrate how to avoid such effect and provide more details to demonstrate the reliability of the human decision collection."

The same original image, corrupted or not, was never shown again to the same participant, for exactly this reason. We’ve added this clarification to the experiment description.

We very much appreciate the valuable comments and questions. We address each of them individually.

"However, some of the top performing robust models (ImageNet-C leaderboard) like DINOv2, and MAE are missing in the evaluation. It is helpful to understand behaviour of these models in VCR."

We have added DINOv2, the top performer on the ImageNet-C leaderboard, to our evaluation. It also performs best in terms of VCR, along with the other two already-included transformers (TianDeiT-A and -B). The identified robustness gaps compared to humans remain though for all three model, especially for blur corruptions, so the paper conclusion remains unchanged. We’ve run into some problems interfacing the existing MAE models with ImageNet, but should have this resolved for the final version of the paper.

"In page 3, under Testing VCR, it is mentioned that “only sufficient data in each group but not uniformity”. Why this is the case? How do you define the sufficient data here? What are the drawbacks of considering uniformity."

Achieving uniformity in VCR requires inverting the mapping from VCR values to parameter values, which would require a costly optimization loop, which is sketched in Sec 8.5. However, since we are fitting a spline, the simple Alg. 1 in Sec 8.5 can tolerate the variation in bin sizes, even if the coverage of the bins is not 100% (see Table 1). This simple algorithm is similar to how average precision for object detection is computed, by first monotonically interpolating the existing datapoints before computing the area under the curve.

"It is mentioned that “ this specific design removes the possibility of biased results”, can you clarify what kind of biased distribution of data is referred here?"

ImageNet-C computes robustness by averaging performance over a set of corruption parameter levels, and it does not fit a continuous function before computing the average. As a result, and because of the non-linear (and varying across transformations) relationship between the corruption parameters and their visual effect (as measured by VQA), the ImageNet-C approach leads to big gaps in coverage of the visual change range, such as covering only low or only high visual change levels (see Figure 11). For example, ImageNet-C has no samples with Delta v below 0.5 for Gaussian noise; see Figure 11e in the appendix. As a result, the robustness error reported by ImageNet-C for Gaussian noise is biased towards the far end of the visual change range. Conversely, the robustness error reported by ImageNet-C for Gaussian blur does not consider robustness in the far end (Figure 11o) and is thus biased towards the low end.

"Humans are presented with one image at a time for 200 ms? Isn’t it too short to notice the image? Are the human participants in an average recognize objects in the image within that time? Would it be safe to assume that human participants do even better job when presented with an image upto 1s? It is mentioned that time was set to ensure fairness. Are the machines classify each image with the same 200 ms?''

We follow an established experimental protocol from Gherios et al. 2019a and Hu et al., 2022. The human visual system (HVS) achieves object recognition with feed-forward processing from a single glance within its early (ventral) stream anywhere between 100 to 200ms (see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3129241/). Above 200ms, recurrent processing starts to play a role, where evidence is integrated over multiple scans of the image (like in test time augmentations), and it may eventually involve functions of the pre-frontal cortex. In that sense, as you point out, human performance will increase with time and the increased recursive processing and the subsequent conscious analysis of the image content in the pre-frontal cortex. We limit the experiment to 200ms to compare the performance of feed-forward DNNs with the feed-forward processing in the brain, where the comparison is fair. This clearly would not be the case when the human subjects start to involve the full spectrum of their brain functions.

(continued in the next post)

We thank the reviewer for valuable comments. We address each weakness and question listed in the review individually.

“Objectives are not clear, and so possible insights that one should gain from reading this work. This makes difficult also to grasp the conceptual contributions or take-aways expected from the experimental analysis and results.”

The key objective is to improve the measurement of robustness and comparison to human performance. The key take-away is that robustness needs to be measured over the full and continuous range of visual change to avoid biased results. The use of a visual quality metric to quantify this range allows standardizing the reference over different transformations, since each transformation has different parameters and effects. We have emphasized the key insights in the introduction and conclusions (see the statements highlighted in boldface in the conclusion).

"It looks also strange that a paper that proposes a dataset of corruptions applied to images does not show images of such corruptions and how they relate with existing benchmarks."

The image corruptions are shown in Fig. 10 in the supplementary appendix. It takes a full page, so there is no space for it in the main paper. We have added a reference to the figure from the main body of the paper. The transformations themselves are from the existing benchmarks, 9 from ImageNet-C, four from Albumentations, and one from the previous study by Gheiros et al., and this is specified on p. 4 of the main paper.

"The paper does not provide insights on how the results should be used: to design new models? train existing architectures differently? or other. While the user-study and experimental analysis is large, there is little to none instructive conclusions."

The main insight is that failing to measure robustness over the full and continuous range, as done in existing benchmarks, leads to biased results that do not fully reflect the actual robustness compared to humans, as supported by the experiments. The key message is: What is not measured properly, cannot be improved. Designing new models or training techniques is not in the scope of this paper.

(continued in the next post)