What Makes ImageNet Look Unlike LAION

Thank you for the review of our work.

Thank you for pointing out the lack of reference about ImageNet's uniqueness. The seminal work of "Unbiased look at dataset bias." (Torralba and Efros) has initiated such studies. We will consider providing further references in the final version.

We agree that LAION and ImageNet are created from different data sources. However, as long as images on Common Crawl and Flickr are sufficiently rich, we can assume they have the potential to contribute diverse images for each class. Therefore, the difference in intra-class similarity should be explained by identifying where this diversity is lost in the selection mechanism.

Thank you for pointing out to DataComp. We believe similar studies including a replication of our work potentially for other benchmarks can be done on DataComp as well and we will suggest this in our updated discussion.

Thank you for a helpful review of our work. In the following, we discuss two main concerns raised in the review: 1) The contribution of CLIP thresholding in the creation of LAION, and 2) the possibility of noisy labels. We will also address other raised concerns in the last section of the response.

1. Contribution of CLIP thresholding in the creation of LAION

Thank you for this insightful question. We agree that image content is utilized in the creation of LAION where images with CLIP image-text similarity of 0.3 or higher are selected. However, in this process, the image content is compared to its caption, limiting the leak of image information to the extent that can be conveyed by the caption. Therefore, such image-text matching can only distort the distribution of images to the extent that the text can reveal information about the image. Given that images are a far richer modality than text, the text still serves as an information bottleneck. Compare this to the case when a human labeler directly examines the images, with all the potential biases and shortcomings they may have in recognizing the concept. Or when a search engine directly acts on image embeddings or uses popularity metrics to return the results. In these cases, the information leak can be much greater than the leak through matching with the caption.

In fact, for the specific selection mechanism we employ, the situation is even better. The worst-case scenario when LAION was created would involve a leak of information from the image beyond the class itself, such as the clarity of the image or the difficulty of identifying the object. Even if such information is present in the text and influences the selection of images into LAION, by intentionally avoiding the search for visual descriptions in the caption, we have grounds to believe that our LAIONet selection mechanism is not exploiting this leaked information and is thus less susceptible to selection bias.

Lastly, while it's conceivable that a leak of image information occurs when LAION instances are selected based on the 0.3 threshold, the degree to which this leak influences the final images is questionable. Notably, this leak occurs in the initial stage of obtaining a candidate pool; however, our final selection is predominantly influenced by the stringent requirement on textual similarity while deliberately neglecting visual descriptions. Our strict criteria for this similarity significantly reduce the dataset size from over 12M to less than 1M, serving as the primary requirement for image selection.

In summary, we have never denied the possibility of an information leak from the image through the text. In our depictions of causal graphs in Figure 2, we explicitly illustrate the bidirectional link between the image and the text. For instance, image information naturally leaks into the text when someone writes a caption describing the image, as was also evident when LAION candidates were filtered based on image-text similarity. The essence of our argument is however that this leak and the potential distortion of the final selected images are limited.

Thank you for your helpful review of our work. We give a detailed answer in the following.

---

This paper still lacks a central objective.

Concerning the objective of our paper, we pose the intriguing question of whether LAION offers images that ImageNet was not capable of and, if so, why. Answering this question comes with new insights about both ImageNet and LAION with potential takeaways for future dataset creation. In summary, our study retrospectively examines ImageNet, pinpointing a selection bias in its data-generating process. This insight was made possible through experimentation with LAION. Additionally, we demystify the distributional advantage of LAION and illustrate how one can leverage an information bottleneck to obtain more diverse representations of a concept.

---

... both Figure1 and Figure5 seem to indicate that model performance on ImageNet and LAIONet is positively correlated. This suggests that LAIONet doesn't offer additional indicative value for model performance analysis, which is typically the most important for classification datasets.

Regarding the consistency of performance rankings in ImageNet and LAIONet, we do not believe such consistency undermines the value of evaluation on LAIONet. For example, the accuracy of ImageNet-trained models assessed on LAIONet provides insights into the extent of generalization lost due to reduced diversity. Additionally, beyond accuracy, the comparison of ImageNet with LAIONet enabled the identification of lower within-class diversity. Therefore, evaluating on datasets like LAIONet, while it may not significantly alter model rankings, can still offer valuable insights into model performance in a new domain and illuminate dataset intricacies. It's also worth noting this consistency is not surprising as the widely-used ImageNetV2 also showed consistent ranking of the models but this does not undermine the new insights it offered.

---

... the ImageNet dataset and the LAION dataset were created at different times and for different purposes.

Concerning different purposes in the creation of ImageNet and LAION, we agree that these datasets are obtained at different times for different learning objectives. However, this comparison has been insightful about both datasets: First, using LAION as an image search engine enabled us to recreate ImageNet in a controlled way and get new insights about ImageNet by contrasting it with the new dataset. This is how we were able to identify a selection bias. Second, large-scale language image models trained on LAION demonstrate unprecedented robustness in image classification. This raises the question of what makes these models so robust. Previous works have conjectured that the LAION image distribution is the primary cause [Fang et al.]. Our work provides explicit evidence that even for the same task and classes, LAION can provide more diverse images than those selected into the classic ImageNet. Thus, such a comparison has been insightful in understanding LAION's capabilities as well.

---

... what specific new recommendations are included?

Regarding recommendations for future data collection, we propose that selection based on an information bottleneck, such as concise text, is a promising method to avoid selection bias. Whenever diversity is desired, this mechanism can be employed to acquire diverse instances from the target class. This class may not necessarily belong to ImageNet classes or other well-known benchmarks. As a concrete example, based on the feedback received for our work, a group of industry-based researchers found our selection mechanism promising for obtaining diverse images for a new concept, enabling them to calculate a more representative image embedding for that concept to be later used as part of a search engine. We also received feedback like ``our team spent a long time curating a dataset and then we realized the improvement on it does not translate to our real application. Now I can clearly see what goes wrong.'' These examples have been heartwarming for us.

---

Considering the different collection times of the two datasets as mentioned above, is the gap in intra-class similarity related to the distributional shift of internet data? Also, given that ImageNet-1K was derived from ImageNet-22K, would an analysis of ImageNet-22K be more meaningful?

Concerning different collection times, we agree this can explain a small part of the significant difference observed between the two datasets. This concern may be more pronounced for technology-related concepts and when images are compared across datasets. However, this is not the case for the majority of ImageNet classes, and we compare image similarity within each dataset rather than across datasets, so, this should be less of a concern.

We also agree a similar experiment can be done on the larger ImageNet and possibly on the larger LAION 5B version however we do not expect this to change the conclusions.

Thank you for the review of our work. In the following, we provide a detailed response to the raised concerns.

---

The only conclusion of this paper is that ImageNet is more of an easy dataset than LAION ... This conclusion is unsurprising since ImageNet is curated very carefully to exclude outlier examples.

Our study stems from the intriguing question of whether for the same task, the recently popular LAION can provide images that ImageNet could not. Indeed, one major finding here is that the ImageNet-like dataset created from LAION, termed LAIONet, demonstrates higher intra-class variability, posing a challenge for ImageNet models to generalize.

We find this observation to be particularly surprising, considering that we created LAIONet in the same manner as ImageNet, using LAION instead of Flickr as the image search engine. If we assume that LAION and Flickr images closely resemble random images from the web, one might expect the resulting dataset, for appropriately chosen thresholds, to resemble the original ImageNet. However, we show that this is far from the case.

As you pointed out, we diagnose the cause of this contrast and provide an explanation in terms of a difference in data-generating processes. More precisely, we identify a selection bias resulting from information leakage from the image at the time of selection. Although the existence of selection bias might be trivial, our work shows this mechanism has been significantly at play at the time of ImageNet creation.

We want to emphasize that the LIONet-ImageNet difference is not caused by noisy labels or outliers within LAIONet. In fact, our conservative approach to creating LAIONet, along with extensive experiments, supports the quality of LAIONet labeling. We have elaborated on this in our response to reviewer fkPT. The difference is precisely due to the selection bias in ImageNet, which has led to a reduction in the intra-class variance of the images.

---

I do not see much value of the findings.

Concerning the potential implications of our findings, our text-based selection exemplifies how a modality with less information can act as an information bottleneck for selecting instances from a richer modality. Our analysis of LAIONet reveals that such a selection mechanism can offer more diverse examples for a given concept. As a concrete example, based on the feedback received for our work, a group of industry-based researchers found our selection mechanism promising for obtaining diverse images for a new concept, enabling them to calculate a more representative image embedding for that concept to be later used as part of a search engine.

Our study also complements previous empirical observations [Fang et al.] that the robustness of contrastive language image models should be caused only by a diverse training distribution. Our work demystifies this distributional advantage and explains what was missing from a conventional dataset.

---

Visual datasets should not be curated only using text descriptions, which leads to a higher probability of getting wrong images inside the dataset.

It's essential to implement safety measures based on specific downstream tasks. In fact, in applications that require applying safety filters based on full image content, as long as the filter is accurate, the distribution of the safe images will remain mainly unaffected and our conclusions remain valid.

---

... the conclusion of this paper only supports curation using the names of classes, and thus has limited values.

As a minor clarification, our text-based selection relies on the unique description of the class as well and the name of the class is not the only identifier.

---

Why not pretrain models on both datasets and compare the differences to support your conclusion?

While training model on LAIONet is feasible, it does not substantially contribute to our conclusion beyond the current results indicating that LAIONet images are more diverse. Consequently, we deemed such an experiment unnecessary and instead directed our focus towards understanding the underlying cause of this difference.