Unveiling AI's Blind Spots: An Oracle for In-Domain, Out-of-Domain, and Adversarial Errors

[YWy5.1-Claims And Evidence] First, our paper serves as a proof-of-concept, showing that training mentors with adversarial attack (AA) errors from the mentee has a greater impact on improving error prediction accuracy than training with in-domain (ID) or out-of-domain (OOD) errors. In Tab. 1, we use the adversarial images produced in the final iteration of the PIFGSM method to train SuperMentor.

As requested by the reviewer, to probe the decision boundary of the mentees, we used adversarial images from all iterations of the PIFGSM method (with 3 iterations in total) to train our SuperMentor and named this new SuperMentor as "En-SuperMentor". By doing so, we can further diversify the samples and better capture the mentee's decision boundary. As shown in Tab. R2 in link, En-SuperMentor boosts average accuracy from 77.0% to 84.1%, which is significantly better than the other baselines and original SuperMentor.

Next, as requested, we also added two quantitative loss landscape analyses using the method in [b]. In these analyses, we apply small perturbations to the mentor's weights and monitor the resulting changes in SuperMentor’s average accuracy.

In the first loss landscape analysis (Fig. R4 in link), we examine mentors trained on three different error types: ID, OOD, and AA from the ImageNet-1K dataset. The results indicate that mentors trained on AA errors exhibit a wider loss landscape than those trained on ID or OOD errors. This suggests that training mentors with adversarial images makes them more robust to weight perturbations, thereby enhancing their ability to capture the generic features for predicting the mentee's decision-making. This finding strongly supports our loss landscape analysis discussed in Sec. 4.1.

The second loss landscape analysis (Fig. R5 in link) examines the mentor’s error prediction performance using two different backbones: ResNet50 and ViT. The figure shows that mentors with ViT architectures have a considerably wider loss landscape compared to those with ResNet50. This further reinforces our claims in Sec. 4.2 that transformer-based mentor models outperform their ResNet-based counterparts in error prediction.

We will add these results and discussions in the final version.

[b] Li et al. "Visualizing the loss landscape of neural nets." NeruIPS 2018.

[YWy5.2-Questions For Authors] The MLP branch associated with distillation loss $L_d$ enables the mentor to mimic the mentee's predictions, and the MLP branch associated with loss $L_r$ ​is responsible for predicting whether the mentee will make mistakes on the input images. Without the first MLP for $L_d$​, the mentor cannot effectively capture the learning patterns of the mentee within its feature extraction backbone. We conducted an ablation study (see Appendix, Tab. S4) by removing the first MLP (i.e., eliminating the distillation loss $L_d$). The experimental results indicate that excluding the first MLP leads to a significant decrease in the mentor’s performance across all datasets.

[YWy5.3-Questions For Authors] The concept of out-of-domain data remains somewhat ambiguous (Guérin, et al., 2023) in computer vision. In our paper, we define out-of-domain data as any data that falls outside the training domain (see Sec. 3.3), following the settings outlined in (Hendrycks & Gimpel, 2016; Luo et al., 2021; Yu et al., 2022). Although corrupted images have the same semantic meaning as clean images, the corruption methods introduce distributional shifts relative to the training domain. Whether these noisy images are classified as ID or OOD is irrelevant; what truly matters is that our SuperMentor can predict the errors of mentees when noise is added to the original images.

[YWy5.4-Questions For Authors] According to the suggested paper by the reviewer, the natural adversarial samples refer to the samples that always lead to wrong predictions regardless of which AI model we use. Following the same definition, we defined sets $N_1, N_2, N_3$ in the caption of Tab. R3 in link. The result in Tab. R3 indicates that the natural adversarial samples in $N_1$ play a key role in explaining SuperMentor’s strong generalization across different architectures. However, SuperMentor can also perform well on the non-natural adversarial samples ($N_2$ and $N_3$​). For example, it achieves 77.7% accuracy on the set $N_3$ for ID error types. This demonstrates that SuperMentor’s robust generalization ability is not limited solely to natural adversarial samples.

We will cite the suggested paper and add these new results in the final version.

[B54K.1-Other Strengths And Weaknesses] Thank you for your insightful comments! We address the points raised as follows:

First, training a mentor to distinguish specific error types is a promising future research direction, as we have mentioned in Sec. 5 of our paper. This task is more challenging than predicting whether a mentee will make an error. It requires a mentor to first achieve high accuracy in error prediction before extending its capabilities to distinguish the exact types of errors a mentee may encounter. Therefore, our work serves as a solid foundation for this promising research direction.

Second, we agree that our method requires access to the mentee's model parameters to generate adversarial images. Training the mentor without using these parameters (i.e., treating the mentee as a black box) presents an interesting and challenging future research direction. However, we respectfully disagree that our mentor needs access to the mentee's training data. Our mentor is instead trained solely on the mentee's behaviors on its evaluation datasets. This is a significant advantage for real-world applications, especially when training the mentee is resource-intensive (e.g., high memory usage) or when the training data of a mentee is inaccessible.

Third, as described in Appendix, Sec. I, we conducted experiments evaluating our SuperMentor on additional OOD domains of the ImageNet dataset, including errors arising from spurious correlations. Specifically, we tested our SuperMentor on datasets such as ImageNet9-MIXED-RAND [a], ImageNet9-MIXED-SAME [a], and ImageNet9-MIXED-NEXT [a], which include images with random backgrounds from a random class, random backgrounds from the same class, and random backgrounds from the subsequent class, respectively. These datasets are designed to reveal how models can misclassify objects by focusing on spurious background cues rather than the objects themselves. For instance, an AI model might label an object as a bird based solely on a tree in the background. The experimental results demonstrate that our SuperMentor can still achieve above-chance error prediction performance on these datasets.

We will include these discussions in the final version of our paper.

[a] Xiao, et al. "Noise or Signal: The Role of Image Backgrounds in Object Recognition." ICLR 2021.

[B54K.2-Questions For Authors] Thank you for the reviewer's insightful question! As mentioned in Sec. 5 of our paper, the reviewer’s point is indeed a promising future research direction. We would like to emphasize that our work serves as an important foundation in this research direction, as mentors with higher error prediction accuracy may enhance the training of mentee models by correcting their mistakes on the fly.

[B54K.3-Questions For Authors] We would like to clarify that the number of samples differs between Joint training and AA training. According to Appendix, Tab. S1, Joint training uses 25092 samples, whereas AA training uses only 6774 samples. The marginal improvement observed in Joint training may be attributed to the larger training sample size. This suggests that the quantity of samples is not the key factor in enhancing a mentor’s error prediction performance. Training mentors on samples that accurately capture the error patterns of mentees is crucial. For instance, the adversarial images generated using the PIFGSM method in our paper effectively illustrate this point. With only 6774 training samples from PIGFSM—one-fourth the size of the Joint training dataset—our SuperMentor on AA training achieves performance comparable to Joint training. These findings further underscore the significance of our study by providing critical insights into optimal training practices for mentor models.

[LEYc.1-Claims And Evidence] We appreciate the reviewer’s thoughtful question and would like to clarify the following two points:

First, our evaluation settings are fair for all mentors. The dataset split presented in Appendix, Tab. S1 applies only to the error source used for training the mentor, not to all error sources. For instance, if Mentor A is trained on adversarial images generated by PIFGSM error source, only the PIFGSM-generated samples are split into training and evaluation sets according to Appendix, Tab. S1. All samples from other ID, OOD, and AA error sources are fully used for evaluation. Thus, comparing its performance with that of other mentors on these error sources is fair. It is true that Mentor A may perform better on the PIFGSM error source since the mentor was trained on it. However, even when excluding its performance on the PIFGSM error source, its overall average performance ranking compared to other mentors remains largely unchanged. This observation applies to mentors trained on other error sources as well. Confusion matrices in Appendix, Fig. S3, S4, and S5 show that the evaluation results on the trained error sources (accuracy in the white background of confusion matrices) do not dominate the average performance (accuracy in the pink background of confusion matrices).

Second, we appreciate the reviewer's question regarding a potential confounding variable in mentor training—specifically, whether AA, OOD, or ID examples inherently provide more predictive information, or if some error examples are simply easier or harder to learn. To address this, we adopted an alternative dataset splitting strategy for mentor training, distinct from the one in Appendix, Tab. S1. For example, we start with the original images from CIFAR-10 and separately apply two perturbation methods, PIFGSM and SpN, to produce two sets of error sources. Next, use 60% of the PIFGSM-generated images to train a mentor model, and reserve the remaining 40% for testing. Similarly, use 60% of the SpN-generated images (corresponding to the same original CIFAR-10 images as the PIFGSM training set) to train another mentor model, and reserve the remaining 40% (corresponding to the same original CIFAR-10 images as the PIFGSM testing set) for testing.

This approach is applied consistently across mentors trained on other error sources, ensuring that all mentors are trained and evaluated on images derived from the same original CIFAR-10 images but different domain shifts. Consequently, the confounding factor related to the inherent ease or difficulty of learning certain samples is eliminated, leaving only the question of whether AA, OOD, or ID samples inherently provide more predictive information.

Figures R1, R2, and R3 in link present the performance of mentors on various error sources for the CIFAR-10, CIFAR-100, and ImageNet-1K datasets, respectively, under this new data split strategy. The results reveal that training mentors with adversarial attack errors from the mentee leads to a greater improvement in error prediction accuracy compared to training with ID or OOD errors. Consistent with our original finding reported in the paper, this finding confirms that adversarial images carry more predictive information than either ID or OOD images.

We will add these discussions and new experimental results in the final version.

[LEYc.2-Claims And Evidence] Thank you for the suggestions! To support our claims in Fig. 3, we computed p-values to compare the error prediction accuracy of mentors trained with different error types. Our primary claim is that mentors trained on the adversarial attack (AA) error type achieve higher accuracy than those trained on in-domain (ID) or out-of-domain (OOD) error types. To validate this, we calculated p-values using a two-tailed t-test and compared the performance of AA-trained mentors to ID-trained mentors and AA-trained mentors to OOD-trained mentors across four different [mentee]-[mentor] settings (ResNet50-ResNet50, ViT-ResNet50, ResNet50-ViT, ViT-ViT) on three datasets: CIFAR-10, CIFAR-100, and ImageNet-1K. All pairwise p-values are below 0.05, indicating that these performance differences are statistically significant. Therefore, our claim is well-supported. We will include these p-value analyses in the final version.

We also agree with the reviewer's suggestion regarding Tab 1. The updated version, now including variance over 3 runs, is shown in Tab. R1 in link. This confirms that the performance differences between our proposed method and the baselines are not simply attributable to noise. We will include this updated table in the final version.