Does Progress On Object Recognition Benchmarks Improve Generalization on Crowdsourced, Global Data?

Thank you for your appreciation of our work, and thoughtful questions. We are glad you found our work to address an important topic, containing interesting and important findings, providing a balanced view, and well-written. We hope to have addressed your questions, and welcome any additional questions or suggestions on how we can strengthen our work:

Difference Between DollarStreet and GeoDE and Figure 8:

There are a few main drivers of the dataset differences from our perspective:

1. DollarStreet is more diverse in terms of geography and household income. DollarStreet was curated to explicitly capture a broad set of households, sending photographers to over 63 countries to people’s homes, and selecting households with a variety of incomes. GeoDE was crowdsourced among internet users from 19 countries and was designed to show the importance of crowdsourced data compared to internet scraped data.
2. GeoDE has a 1-to-many label mapping, whereas DollarStreet has a 1:1 label mapping. As shown in Appendix A.1 , GeoDE classes are coarser as there are GeoDE classes that map to as many as 25 ImageNet classes, while DollarStreet has 1 ImageNet label for each DollarStreet label. This makes GeoDE an easier dataset than DollarStreet for ImageNet models (we also evaluate a large number of zero-shot models that use the ground truth labels, without requiring mapping).

Despite the large differences in dataset curation and difficulty, we find very similar results on DollarStreet and GeoDE, indicating that the problems we discover are not specific to a given dataset curation, or labeling. We find that both datasets have large geographic disparities, even with SOTA models. Most critically, we find for both benchmarks that disparities are not resolved by progress on standard generalization benchmarks, dataset scaling, standard robustness interventions, or architecture scaling.

GeoDE in Table 2:

We included only DollarStreet in Table 2 in order to avoid confusion in reader interpretation of our Progress gap measure, which is a ratio of the rate between the benchmarks with respect to one baseline (DollarStreet). We find similar trends hold, showing conclusions are consistent across both datasets - the progress rate on GeoDE is smaller compared to other benchmarks. We have these numbers in Figure 5 in the Appendix and would be happy to add them to the main table if helpful.

Income Variation in DollarStreet:

We agree that controlling for income variation is very interesting, thank you for suggesting this analysis! We have run an analysis of OpenAI’s CLIP trained on LAION 2B, analyzing the per-region accuracies within each income quartile. For the middle two quartiles (the only quartiles with both Europe and Africa data), we found that there was still a substantial performance drop across regions. This indicates that even when controlling for income, there is a significant performance degradation across geographies. We have included the table in Appendix C, and below for reference - thank you for the excellent suggestion, and we welcome any further advice to clarify this point.

Thank you for your thorough engagement with our work and thoughtful suggestions! We are particularly glad that we were able to convey the very pertinent problem of geographical disparities, and are glad that you found the insights in our work important. We hope we addressed your questions thoroughly below, and welcome any additional questions!

Driving Mechanism of Geographical Disparity:

Our work primarily focuses on highlighting geographical disparity as an understudied and open research challenge for the community by providing comprehensive empirical evidence that this problem is pervasive across architectures and training paradigms. While the exact underlying mechanisms for models’ disparities across geographies (as well as a solution to this problem) is an open question, below we outline our main hypotheses for the driving reasons behind these disparities:

1. Dataset imbalance: Our work shows that the geographical disparities are persistent across 100+ models with a variety of training architectures and paradigms. This suggests that the disparities are driven by geographical data imbalance in pretraining datasets. Moreover, prior works showed ImageNet is heavily western biased [1], and in our paper Appendix F we present a kNN clustering experiment, where we analyze CLIP embeddings of LAION and DollarStreet to approximate the regional distribution of LAION. We find that our proxy measure indicates an extreme imbalance in LAION, which aligns with existing work on ImageNet. Additionally, in Section 6.3 we show that fine-tuning the last layer of a ViT model on DollarStreet improves disparities on both DollarStreet and GeoDE, showing the promise of interventions using geographically diverse data.

2. Distribution shifts and factors of variations between geographies: Gustafson et al [2] annotated factors of variation in DollarStreet and found that variations in texture, occlusion, and lighting were among the factors most associated with performance disparities. Models pre-trained on ImageNet may overrely on these factors and other spurious features such as backgrounds [3] which also contributes to geographical disparities.

3. Household income: In addition to geographical labels, DollarStreet also has income group labels, which is correlated with geography. We hypothesize that the lower income group is underrepresented in pre-training data and also presents a distribution shift in terms of factors of variations discussed above.

Importantly, scaling up model or dataset size as well as standard robustness interventions such as data augmentation, while being effective on standard ImageNet OOD benchmarks, don't lead to improvements in geographical generalization, highlighting the unique challenges of geographical distribution shifts. We add this discussion as a section in Appendix I.

[1] Shankar et al, No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World [2] Gustafson et al, Pinpointing Why Object Recognition Performance Degrades Across Income Levels and Geographies [3] Xiao et al, Noise or Signal: The Role of Image Backgrounds in Object Recognition

Label Mappings and Label Noise:

Thank you for the insightful question! We also considered label noise as a potential concern, which is why we explicitly conduct zero shot classification experiments, where we prompt a model with the original DollarStreet labels rather than relying on an ImageNet mapping. Specifically, we evaluate a suite of CLIP models (See Appendix A for list). We find zero shot models have consistent geographic disparities on both DollarStreet and GeoDE benchmarks, finding some of the largest disparities for the largest CLIP models, even with scaling (see Figure 2). We would also like to emphasize that in both cases with DollarStreet and GeoDE, the original labels given are much more coarse than ImageNet (for example: ‘bag’ label compared to ‘purse, pencil case, backpack, etc.’), thus when performing label mapping to ImageNet-1K each example DollarStreet and GeoDE is assigned multiple plausible ImageNet labels. This adds additional protection against label noise, as the model's prediction is considered correct if it falls within any of the more granular ImageNet categories. We also added a replication of Figure 5 with DollarStreet’s Top 5 accuracy (rather than Top 1), finding the same result: that our rate of improvement on DollarStreet lags significantly behind progress on standard generalization benchmarks.

Thank you for your appreciation of our insights as important to the community, and especially relevant in the era of foundation models trained on large-scale web data. We are glad that you found our results interesting!

Our primarily empirical analysis as limited:

We agree that our work is largely empirical. However, we disagree that empirical work has limited contributions. We believe that some of the most impactful work in machine learning has been work that provides scientific insight, and redirects the field to new problems, new model limitations, or new approaches . Our intention with this work is to highlight geographical disparities as an open and practically important scientific challenge for the research community, and we support this with actionable insights below

1. Our work shows for the first time how extensive geographic generalization failures are across model architectures, training paradigms, and research datasets.
2. We discover that these failures, unlike those represented in existing benchmarks and counter to researcher intuition, are unsolved by current mitigations, data scaling, or model scaling.
3. These results indicate that standard measures of progress have over-indexed on ImageNet and ImageNet-based metrics, leaving geographical disparities as an understudied and critical problem affecting real-world deployment.
4. We offer new ways to both measure and mitigate this problem, with our KNN rebalancing experiment, and last layer retraining experiments. We also release a simple-to-use benchmarking library allowing researchers to evaluate models with just 4 lines of code.

Discussion on Dataset factors and selection:

Thank you for this insightful suggestion! We absolutely want to emphasize the importance of data selection as a main takeaway of our work. We added some language to our discussion and an additional section in Appendix H describing these factors to our paper. To your specific questions:

1. Would dataset diversity be an important factor in evaluation? Absolutely! Our empirical evidence indicates that geographical diversity is a missing axis from standard benchmarks, and we hope to encourage the field to consider geographical disparities (and all data subgroups) as important factors for evaluation.
2. What factors should be considered in designing proper evaluation benchmarks? Our work provides evidence that data sources, label curation, geographic representation, and income information should all be closely considered. In Appendix H, we emphasize that the best evaluation metrics are crowdsourced and grounded in real-world use, which is a significant difference between standard benchmarks and our geographic benchmarks.
3. What are the right evaluation metrics to consider for evaluation foundation models? We address this question by highlighting the need for using subgroup evaluation metrics in generalization literature, and the limitations of using average accuracy alone. We will cite relevant examples of subgroup analysis in fairness and spurious correlations. Similarly, our work highlights the impact of looking more closely at the kinds of accuracy gains made (improvement on which portions of data), which we demonstrate with our progress gap measure. Overall, we hope to suggest that there is no one particular benchmark or tool that can perfectly measure model reliability, but that having a comprehensive suite of measures, and by analyzing benchmarks more precisely, we can improve our understanding of model behavior and its reliability.

Thank you for your insightful suggestions! We welcome any other questions or suggestions to improve the clarity and impact of our work.

Thank you for your thoughtful appreciation of our work! We are glad that we were able to convey the impact of our empirical contributions, the novel insights of the geographical dataset imbalances we characterize, and the crucial implications of our results for benchmarking.

Using 2 Benchmarks for Evaluation of Geographic Disparities:

Our analysis does rely on two benchmarks to characterize geographical disparity, and we agree that our results (as with all research analyses), should be interpreted considering the datasets used. However, we emphasize that these benchmarks are reliable measures of geographical disparities, as they are 2-6x larger than existing robustness benchmarks, and are supported by a wealth of research use in fairness [1,2,3]. DollarStreet and GeoDE have 61K and 38K respectively, compared to 10K in ImageNet-A and 30K samples in ImageNet Sketch. We have added a paragraph in the Appendix A (in blue) to clarify our use of these benchmarks, and welcome any further suggestions.

[1] Shankar et al, No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World [2] Gustafson et al, Pinpointing Why Object Recognition Performance Degrades Across Income Levels and Geographies [3] Xiao et al, Noise or Signal: The Role of Image Backgrounds in Object Recognition

Balanced Retraining on LAION:

Thank you for this insightful suggestion! In our work, we demonstrate how geographical disparity has worsened with progress on standard benchmarks, is exacerbated by data and model scaling, and produce some of our preliminary evidence of dataset imbalance in LAION as a contributing cause. We agree that the next step in this research direction is to show how our insights can be used to design rebalancing mitigation strategies for large-scale training datasets. We find this suggestion of performing a rebalancing from our LAION cluster analysis, and are excited to try it. This is quite a computationally demanding experiment, which requires balancing a very large scale dataset, coordinating distributed training, etc. which is why we did not explore a rebalancing mitigation in the original submission. However, as we wrote in Appendix F, our clustering analysis should be viewed as a rough proxy of the regional distribution, as there are significant limitations - most notably, that LAION has several orders of magnitudes more concepts represented than DollarStreet. Therefore, rebalancing LAION based on comparisons to DollarStreet may not balance the dataset well across concepts. We are excited to try this however, and will share results when we have them.

Contribution of Last Layer Retraining:

We agree that last layer retraining has certainly been used in other works, which we cite in Sec. 6.3. In this work, we demonstrate empirically that geographical disparities are a pervasive problem, and that existing methods, such as scaling and standard robustness interventions, fail to consistently improve it (Figure 3 and Table 3). Our intention with our experiment in 6.3 is not to claim the development of a new method itself, but to show how, counter to these other approaches, last layer retraining could be a practical mitigation for improving geographical robustness. We expand on existing work by showing how last layer retraining is not just allowing adaptation to a given geographic distribution ([1]), but that last layer retraining improves geographic robustness on new datasets (without requiring the monumental task of generating labels for web-scale data). We have added some (highlighted) language in Sec. 6.3 to clarify.

We welcome any other questions or suggestions to improve the clarity and impact of our work!

1. Vikram V Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. Beyond web-scraping: Crowd-sourcing a geographically diverse image dataset. arXiv preprint arXiv:2301.02560, 2023.12

Thanks author for the detail reply and providing additional experiment. I am satisfied with where this paper is heading toward thus change my rating to 6 and soundness to 3. In your future revision, please make sure to include result of retraining CLIP on balanced LAION dataset, and publicly release the label for geographical property of each LAION sample. Also you should include a link in the main text(where you mentioned to your comment on 'Benchmark Use' in appendix.