CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis

We thank the reviewer for many insightful comments. We answer the questions in what follows. Please let us know if further clarification is needed.

Q1: Can you elaborate on the potential biases introduced during data collection from search engines or how did you ensure the collected data is diverse and representative? Lacking a discussion on potential biases and limitations introduced during data collection from search engines. Biases in search engine results can affect the diversity and representativeness of the dataset, which should be acknowledged and addressed.

Thank you for your insightful comments. This paper has provided some related discussions, such as distribution bias between CIFAR-10-Cs and CIFAR-10-W and the influence of noise data, in the submitted paper. Following the suggestion of the reviewer, we would like to provide additional explanations and discussions.

Potential biases: We agree with the reviewer that the dataset searched from search engines may be biased to the target sets and sets from different engines. However, our primary aim is to incorporate a wide range of distributions to evaluate the AccP and DG methods effectively. Hence, while these biases exist, they offer diverse perspectives for our evaluation, aligning with our objective to cover various distributions.

Diverse and representative: The diversity of our collected datasets has been demonstrated through the results in our paper. For example, Tables 2 and 3 show the diversity of datasets by presenting varying accuracy prediction and DG results on the CIFAR-10-W. Fig. 2 and Fig. 5 show statistical distributions of different outcomes across the datasets. Each data point represents results on one set, indicating the variety within our data. Regarding representativeness, we try to follow the natural distribution of each search engine without imposing additional restrictions. We believe that distributions mirroring real-world scenarios are invaluable for comprehensive evaluations.

Potential limitations: Noisy data should be one of the most important limitations of collecting data from search engines. Section 5 has provided some discussion about it. To remove noisy data, we have asked human annotators to review data for several around, and then use the CLIP model to select out data with multiple labels together with human double-check. By doing this, we think most of the noise data has been removed. On the other hand, dataset imbalance may be the other limitation of using search engines because the databases of different engines are different. The balanced distribution of data in the dataset is similar to the data distribution on the engines. At the same time, we discussed the results, such as that shown in Fig.4 (A) and Table.8 (Appendix), of the data being sampled to be balanced in Sections 3.3 and E.

We will incorporate the aforementioned discussions into our paper to enhance the overall understanding of the proposed CIFAR-10-W dataset.

Q2: How might the findings from CIFAR-10-Warehouse generalize to other larger-scale datasets or tasks beyond image classification? CIFAR-10 is indeed a relatively small dataset with low-resolution images and a limited number of categories compared to larger-scale and more diverse datasets like ImageNet or the Wilds[1]. This work does not extensively discuss how the findings from this dataset can be applied to real-world scenarios or other domains.

We agree with the reviewer that CIFAR-10 is a relatively small dataset compared with some existing datasets, such as ImageNet. However, it is one of the most commonly used data in the community and is an important benchmark for AccP and DG algorithm evaluation. Our choice of CIFAR-10 is because of its simplicity and manageability, which makes it a suitable starting point for foundational research of AccP and DG tasks.

We start by using CIFAR-10 as a basic reference to build CIFAR-10 to solve the limitation that existing test data have a small number of domains or use synthetic data for evaluation. Despite its limitations, the insights gained from CIFAR-10 can provide valuable preliminary understanding, which can be a foundation for analyzing more complex datasets. The findings of AccP and DG observed in CIFAR-10 can offer indicative trends that might be observed in larger datasets. For example, if a method cannot work well on the CIFAR-10 setup, it is hard to imagine it works well on a dataset with a large number of categories.

In future work, we will try to build testbeds with datasets of more classes. We will include the above discussion in our revised paper on the potential application of our findings from CIFAR-10 to other real-world scenarios and more diverse domains.

Q3: This work could be strengthened by discussing potential real-world applications beyond the scope of image classification. Evaluating and extending the methods on datasets with different characteristics and applications would provide a more practical significance for real-world applications.

Thank you for your kind suggestion. Indeed, while our current research primarily centres on image classification, there is a large potential for applying our findings to a wider range of real-world applications, such as AccP and DG on object detection and image retrieval. Furthermore, we will consider conducting studies in our future work on these applications. In future revisions of the paper, we will add discussions about other potential applications.

Q4: This paper could benefit from including additional state-of-the-art baseline methods, e.g., GVRT[2], and VNE[3], for a more comprehensive comparison.

Thank you for this valuable suggestion. In response to your feedback, we have conducted additional experiments using the VNE method. The results are as follows, indicating VNE's performance across various subsets of the CIFAR-10-W dataset:

Due to the time constraints of the rebuttal period, we were unable to complete experiments using GVRT. However, we have cited both GVRT and VNE (Min et al.,2022; Kim et al., 2023) as important baselines in our paper In the revised version, we will incorporate a comprehensive analysis that includes GVRT, further enriching our comparative study.

Q5: Can you discuss and provide examples of failure modes for the domain generalization and accuracy prediction methods on the CIFAR-10-Warehouse testbed?

Thank you for this valuable suggestion. Our analysis of the CIFAR-10-W testbed has indeed shown new challenges encountered by both DG and AccP methods.

As evidenced in Tables 2 and 6 of our paper, there is notable variability in the performance rankings of AccP methods when comparing CIFAR-10-W with CIFAR-10-Cs. Specifically, the rank correlation for models such as ResNet44, RepVGG-A0, and ShuffleNetv2 is only 0.481, highlighting a significant discrepancy in evaluations between CIFAR-10-Cs and CIFAR-10-W. An example of this is the performance of the Pred. s (0.9) method. While it ranks as the best AccP method for ResNet44 on CIFAR-10-Cs, it fails to maintain its superiority on CIFAR-10-W. This suggests that the generalizability or adaptability of Pred. s (0.9) may fail when transitioning from CIFAR-10-C to the more diverse CIFAR-10-W environment.

To study the potential failure modes, we first compare the test performance on single source setting and multiple source setting. We observe that when multiple distinct source domains are available at the training stage, it is more likely to achieve higher improvement than that of a single source domain. Furthermore, we compare the test performance of ERM and SagNet under a three source domains regime. In the three source domain training setting, we have two diffusion model-generated domains and a photo-realistic domain. By analysing the testing performance on three meta-categories (i.e., diffusion, KW, and KWC), we find that SagNet brings improvement on KWC (approx. +0.56% on average), but degrades the ERM result on KW (approx. -0.59% on average). This indicates that some domain generalization methods may be able to improve the generalization power of the model on domains that are largely different from training distribution, with the price of reducing some generalization power on similarly distributed domains.

We will incorporate these discussions into the revised version of the paper and add additional discussion on the failure cases of the DG and AccP methods on CIFAR-10-W, providing valuable insights into their limitations and areas for improvement.

Q6: A discussion of potential future directions, and how the dataset can evolve would be insightful.

We appreciate your insightful suggestion. In our paper, particularly within Section 5, we have provided some discussions about the potential impact of CIFAR-10-W in various fields, including the challenges of learning from noisy data. Building on this foundation, and considering the reviewers' feedback, we recognize the critical importance of developing specialized datasets, particularly in fine-grained areas. For instance, the development of fine-grained datasets, as well as those covering a broad spectrum of categories, would be highly beneficial. These datasets are essential not only for addressing specific research queries but also for fostering innovation by introducing more complex and diverse scenarios for model evaluation and testing. We will add further discussion on these potential future directions and the importance of specialized datasets in advancing the field in Section 5 of our paper.

Q3: Are the performance comparison on CIFAR-10-W and existing datasets aligned? and Are there any contradicted conclusions or new observations based on the experiment results on CIFAR-10-W?

Thank you for this important question. Indeed, our analysis reveals some inconsistencies in the performance of methods when evaluated on CIFAR-10-W as compared to existing datasets like CIFAR-10-Cs. For instance, Tables 2 and 6 present differing rankings of AccP methods across these two datasets. Specifically, for AccP, we observed the following rank correlations across different models:

These correlations suggest a lack of strong rank consistency between performances on CIFAR-10-W and CIFAR-10-Cs. This is especially evident for ResNet44, where the correlation is notably low.

Moreover, compared with other domain generalization benchmark datasets (usually involving less than six domains), our proposed benchmark dataset empowers researchers to conduct more statistically meaningful analyses with 180 domains. Specifically, from Table 3, we can see that with the increased number of training domain, the test performances of model tend to be more stable. For example, the variance of test performance drops from approximately 3.9% for single source domain to 0.67% for four source domains. This outcome contrasts with results reported in other studies, indicating that CIFAR-10-W may present a more challenging and realistic testing environment by involving more test sets from different domains.

These observations suggest that conclusions drawn from CIFAR-10-W could differ from those based on existing datasets. In particular, CIFAR-10-W tends to magnify performance differences among methods, providing clearer insights into their robustness for AccP. For DG tasks, the lack of noticeable improvement over the ERM baseline in our dataset suggests that existing DG methods may need further refinement to handle the diverse and complex scenarios presented by CIFAR-10-W.

In summary, CIFAR-10-W serve as a new diverse benchmark dataset, offering a wide array of test environments that are helpful for future research in AccP and DG methodologies. We will include the discussions highlighted above in the revised version of our paper, further indicating the relevance and utility of CIFAR-10-W for future studies.

We thank the reviewer for many insightful comments. We answer the questions in what follows. Please let us know if further clarification is needed.

Q1: Why does this paper divide different domains based on color and cartoon/no cartoon?

Great question. When dividing different domains, various factors can be considered, such as style, color, search engine features, and collection location. Our decision to focus on color and cartoony styles was driven by below reasons:

1. Color: we chose to use color as a distinguishing factor because all the search engines we utilized come equipped with color filtering options. This functionality allows us to naturally leverage these pre-existing data divisions for our analysis.

2. Cartoony Style: Cartoony images represent one of the most common styles beyond natural images. This makes them a valuable category for our study, providing a clear contrast with more realistic image styles. Meanwhile, there are also a large number of cartoon images on the Internet, making it easier for us to collect data.

Furthermore, to diversify our dataset, we also used diffusion models to generate additional datasets in different styles, such as oil painting and sculpture. This helps enhance the range and depth of our domain-based analysis.

We will integrate these discussions into our paper to provide a comprehensive understanding of our current findings. Additionally, will expand CIFAR-10-W in our future work to encompass a broader array of diverse distributions, enhancing its utility.

Q2: Providing more empirical analysis on the advantage and difference of the proposed dataset compared to existing datasets.

Thanks for your kind suggestion. In our paper, we have provided experiments and empirical analysis for comparing CIFAR-10-W with CIFAR-10-Cs when they are used for evaluation. For example, Table 2 shows the results of using CIFAR-10 Cs and CIFAR-10-W respectively for AccP method evaluations, which are analysed in detail in Section 3.2. Fig. 2 (B) shows distributions of ground-truth accuracy and MAE values of method MS-AoL on 180 sets of CIFAR-10-W and 188 sets of CIFAR-10-Cs, respectively. we can observe that CIFAR-10-W has a wider range of dataset accuracy than CIFAR-20-Cs.

During the rebuttal, we have provided new analyses in Section A of the Appendix. Fig. 7 shows the correlation between model accuracy and FD for both CIFAR-10-Cs and CIFAR-10-W. Notably, CIFAR-10-W displays a broader range of FD values compared to CIFAR-10-Cs, indicating a larger variation in dataset distribution. For instance, CIFAR-10-W includes more sets with FD values exceeding 10. Moreover, the stronger correlation coefficients for CIFAR-10-Cs ($\rho = -0.93$ and $\tau = -0.77$) compared to CIFAR-10-W ($\rho = -0.81$ and $\tau = -0.62$) suggest that the distribution of CIFAR-10-Cs, which comprises simulated datasets, is less complex than the real-world data variant found in CIFAR-10-W. We also add t-SNE visualizations of variation within CIFAR-10-W data by color and by the search engine in Fig. 8, which shows that CIFAR-10-W provides a rich and complex dataset that extends beyond the variability of CIFAR-10.

In addition to the above new content, we will try to provide further analysis to enrich our paper with additional insights after the rebuttal process.

Q5: Significant drop in performance in Baidu and 360. Is it due to a lesser number of images or because of the domain gap? I suspect those engines have fewer images which might be causing the accuracy drop.

We appreciate the reviewer's point, but our analysis suggests a different conclusion. The additional results presented in Table 8 of the Appendix, which are based on the CIFAR-10-W balanced subset, show that performance drops for Baidu and 360 remain significant even after adjusting for dataset size. This indicates that the number of images is not the primary factor affecting performance.

To further substantiate this, we calculated Kendall's rank correlation coefficient between the number of images from each engine and the performance of various methods. The average correlation was found to be 0.3730, which suggests only a weak relationship between dataset size and performance across different engines. For domain generalization (DG), the Kendall's rank correlation is similarly low at 0.2355 when comparing the number of images to the standard deviation (SD) of results, reinforcing the conclusion that the performance drop is not due to the quantity of images.

We will include the above-detailed discussion of these analyses in the revised version of our paper to clarify the factors contributing to the differences in performance.

Q6: Separating the search engines in Figure 1 would be have been very informative for a reader, alongside more analysis of drift.

Great suggestion. We have added a new figure of statistics on search engines in Fig. 6 of the APPENDIX. Meanwhile, some analysis of data from different engines is provided. We believe these additions will enhance the reader's understanding of the dataset's composition and the role of different search engines in contributing to its diversity.

Q7: The benchmark would be concerning if there are only 36 visually distinct domains.

We appreciate the reviewer's perspective but would like to offer a different viewpoint. We believe that there are more than 36 visually distinct domains in our dataset. This is because, for each color category, the data from different search engines show significant diversity. We visualize the distribution of different colors and engines during rebuttal. The t-SNE visualizations shown in Fig. 8 of the Appendix clearly demonstrate that the distribution of these images is quite varied, exhibiting a range that is broader than what we observe in CIFAR-10. This diversity justifies our decision not to merge data of the same color from different engines.

Furthermore, the color schemes across different engines are not uniform. To maintain objectivity and avoid the introduction of human bias, we chose not to manually merge data of the same color from various engines in CIFAR-10-W.

Regarding the number of domains, we do not necessarily consider one option—180 or 36 domains—superior to the other. Each approach provides insights into different aspects of the methods being evaluated. We use 180 domains to keep the real-world data distribution native and to prevent the introduction of any subjective bias. Using 36 domains may be able to provide more complex domains for evaluation.

We will add the above discussion to our paper and will explore different domain division ways in our future work.

Q8: What all components would be released publicly?

Thank you for your question. We will release all key components of this paper to the public, including 1) the datasets of CIFAR-10-W, 2) all training sets used in Domain Generalization evaluations, 3) the codes for all the methods, and 4) all the models evaluated for AccP in the paper.

We thank the reviewer for many insightful comments. We answer the questions in what follows. Please let us know if further clarification is needed.

Q1: Why is MAE of prediction scores the main metric reported in the paper for comparing methods in Table 2? The reason for the concern: Fig. 9 shows there is no correlation between MAE and accuracy, while Fig. 7 shows that prediction scores have a high Spearman rank order correlation with ground truth accuracy.

Thank you for your question. In the Accuracy Prediction (AccP) task, existing research often employs Mean Absolute Error (MAE) or Root Mean Square Error (RMSE) as the primary metrics. Consistent with this norm, we use MAE in our paper. The objective in AccP is to predict the classifier accuracy $\hat{Acc}_i, i = 1, \ldots, n$, across $n$ given unlabeled sets. The MAE, defined as

effectively indicates the gap between predicted accuracies and the ground-truth accuracies.

There is a misinterpretation regarding the results presented in Fig. 2 (also in Figs. 7 and 9). Fig. 2 (A) illustrates the correlation between prediction scores $\hat{\text{Acc}_i}$ and actual accuracy $\text{Acc}_i$ Fig. 2 (B) illustrates the relationship between the actual accuracy $\text{Acc}_i$ and the prediction error $|\text{Acc}_i - \hat{\text{Acc}}_i|$ of the AccP method on different datasets. It is expected that there may not be a clear correlation, as the value of prediction error is not necessarily directly influenced by the ground-truth accuracy of a dataset.

We hope the above explanation can help the reviewer have a clear understanding of MAE.

Q2: Conclusions made from task 1 need improvement

1) This benchmark is a harder benchmark compared to CIFAR-10-Cs. The increasing hardness of the testbed seems weird. It is more interesting to highlight whether networks don't capture certain aspects introduced here.

We agree with the reviewer that the goal of building a new testbed is not only to increase the difficulty of the dataset but to propose test data that are more indicative of algorithm performance. The primary aim of developing CIFAR-10-W is to offer a testbed that better reflects the complexities in real-world scenarios and provides a more comprehensive evaluation of algorithms.

Specifically, the proposed CIFAR-10-W can provide a more comprehensive evaluation of AccP methods by including more challenging sets. The challenge here not only means that the data set is more difficult but it is reflected in many aspects, such as more diverse data and more different predicted models. Meanwhile, the evaluation on CIFAR-10-W datasets indeed shows more obvious performance differences of existing methods, which is helpful for highlighting the performance of different AccP methods.

For example, the ranges of y-axis of Fig. 2 (B) show that CIFAR-10-W have more diverse test sets for AccP algorithms evaluation, which helps indicate the robustness of AccP methods under variance environments. Meanwhile, the evaluation of diverse models provides a more accurate comparison of AccP methods when facing different target classifiers. For example, the standard deviations (SD) of different methods for ResNet44 on the CIFAR-10-Cs and CIFAR-10-W datasets, respectively, are 0.81 and 1.80. Obviously, the results difference can be magnified on CIFAR-10-W.

In conclusion, CIFAR-10-W is not only a hard testbed, it is a more diverse and indicative testbed. We will include the above discussion in our revised paper.

2) One can easily argue that predictions are consistent here because the shift is not varied – it simply changes colours rather than the diverse, varied corruptions studied in CIFAR-10-C benchmarks.

Thank you for your comments. We respectfully disagree with the comment that the observed consistency is due to a lack of varied shifts in the data.

In our paper, Fig. 2 and 5 illustrate the distribution of ground truth accuracy for ResNet44 and ERM accuracy on CIFAR-10-W. These figures clearly demonstrate varied performance across our datasets which can indicate the varied shifts among datasets.

Additionally, during the rebuttal phase, the newly added Fig. 7 in the Appendix highlights the distribution differences between datasets from CIFAR-10-Cs and CIFAR-10-W by showing their FD with the CIFAR-10 dataset. It is evident from this analysis that CIFAR-10-W has more datasets with Fréchet Distance (FD) values greater than 10, suggesting a greater deviation from the CIFAR-10 dataset. In contrast, most datasets in CIFAR-10-Cs exhibit smaller FD values, indicating closer similarity to CIFAR-10. Meanwhile, the newly added Fig. 8 in the Appendix shows t-SNE visualizations of CIFAR-10-W and CIFAR-10, which indicates the diversity of images from the same color or different search engines.

Furthermore, we have calculated the rank correlations of different method performances on two subgroups of data from CIFAR-10-W:

There is no obvious correlation, which indicates the variance between these datasets. As for the consistency we mentioned in the paper is caused by the average statistics on all datasets. When methods are evaluated on a large number of diverse datasets, the rank of performance of methods will gradually be stable.

Following the valuable suggestions of the reviewer, we will revise our conclusions and incorporate the above-discussed points into the revised version of our paper.

Q3: Little analysis of the dataset itself, far more focus on task/models

Thank you for your constructive suggestion. In response to your feedback, we have enriched our paper with more analysis of the CIFAR-10-W dataset. The new contents are in Appendix Section A.

Specifically, we have included comparisons of distribution ranges between CIFAR-10 and CIFAR-10-W, highlighting the diversity present within CIFAR-10-W. Additionally, we present an analysis of the variance within datasets categorized by a single color option, illustrating the intricate differences that exist even within seemingly similar groupings.

We believe this added analysis will provide readers with a more comprehensive understanding of the CIFAR-10-W dataset, showing its complexity. This information should further clarify the relevance and applicability of CIFAR-10-W for AccP and DG evaluation.

Q4: How many visually distinct domains exist within these 180 subsets? Specifically, how different are images sampled from different search engines?

Thank you for this insightful question. To illustrate the visual distinction between domains, we have included a t-SNE visualization in Fig. 8 (B) of the Appendix. This figure displays the distribution of bird images from CIFAR-10-W in relation to those from CIFAR-10. The t-SNE plot reveals that CIFAR-10-W encompasses a substantial diversity of images. Each search engine within CIFAR-10-W contributes uniquely distributed data, indicating that the images sampled from different engines are indeed distinct. The spread of data points in the visualization reflects the individual characteristics of each engine's dataset, thereby suggesting that a considerable number of visually distinct domains exist within these 180 subsets. This diversity is critical for evaluating the robustness and generalizability of computational models across a wide range of real-world scenarios.

We thank the reviewer for many insightful comments. We answer the questions in what follows. Please let us know if further clarification is needed.

Q1: One typo: there is an unfinished sentence at the last line of page 6.

Thanks. We have corrected the errors in the revised version and double-checked the paper to avoid typos.

Q2: Is it possible to build finer-grained dataset like CIFAR-100-warehouse?

Thank you for your insightful question. We think it is feasible to build a finer-grained dataset like CIFAR-100-warehouse, but it will face increased challenges and need more effort. For example, if we use the way of building CIFAR-100-warehouse to build a finer-grained dataset, we will face challenges:

1. lower quality web tags, as fine-grained data often require more specialized knowledge.

2. more data cleaning efforts compared to generic classification tasks due to the fine-grained classification.

Despite these obstacles, we believe the construction of such a dataset is achievable. We will consider this in our future work.