Understanding Bias in Large-Scale Visual Datasets

We thank the reviewer for the insightful review and the constructive comments. We would like to address your concerns below.

w1: Even though the paper identifies various factors that help in distinguishing datasets, it does not provide any information/ takeaways on how this information could be used to 'build more diverse and representative datasets in the future'.

Thank you for your suggestion. We are happy to include more discussion about takeaways for dataset curation. Our study provides a general framework for identifying the concrete form of low-level and semantic bias in large-scale datasets. The identified dataset bias can be used retrospectively to analyze the dataset curation procedure. For example, on YCD:

• YFCC contains predominantly outdoor scenes and human interactions.

YFCC samples images solely from Flickr, a platform for user-uploaded photos. As Flickr users primarily share personal photos, landscapes, and social interactions, YFCC images predominantly feature natural scenes and human activities. Moreover, YFCC excludes photos labeled as “screenshots” [1], reinforcing the focus on human-related and natural imagery.

• DataComp has the lowest number of unique objects per image.

DataComp filters for images with high embedding similarity to ImageNet training examples [2], most of which feature object-centric images. While this empirically leads to higher zero-shot performance of downstream CLIP models, it biases the dataset toward images with lower per-image object diversity.

• CC and DataComp are significantly brighter and contain more digital graphics and object showcase.

These datasets are collected from the Internet and feature results from search engines [2, 3], which prioritize professionally created content like advertisements, infographics, and digital media. This results in a higher prevalence of digital graphics and brighter images, optimized for visual engagement and online presentation.

w2: No information is provided on how the accuracy of pre-trained models affects the inferences or findings.

To address your concern, we reran some of the experiments in Sections 3 and 4 using different pre-trained models. While the accuracy of the pre-trained models slightly affects dataset classification accuracy on the transformed datasets, our results and insights remain unchanged.

For Section 3, we use (1) a weaker VitDet-Base [4] model to extract bounding boxes, (2) a weaker SAM (ViT-B) [5] model to extract object contours, and (3) a stronger generative model SD-XL [6] to text-conditionally generate images. Even when using different pre-trained models for transformations, the dataset classification model still achieves high accuracy, suggesting that semantic differences and object shape variations are important contributors to the bias among YCD.

(Due to resource constraints, the SAM experiments are on 300K training samples.)

For Section 4, we use Claude 3.5-Sonnet [7] and Llama 3.1-8B [8] to derive patterns for each dataset as in Figure 16. The dataset features summarized from these two LLMs closely resemble those of the original GPT-4o. Specifically, for YFCC, the emphasis is on "people," "outdoor," and "nature," while for DataComp, the focus is on "white background" and "object focus."

Claude 3.5-Sonnet:

Llama 3.1-8B:

We will conduct more experiments with other segmentation models and object detection models and add the results and discussion to our draft.

w3: There is no discussion on how the format of data transformation affects the findings.

Thank you for pointing this out. We acknowledge that the impact of the transformed images’ format could benefit from further elaboration and a more focused discussion in Section 3:

• Semantic segmentation and object detection: Semantic segmentation provides fine-grained per-pixel semantic annotation, whereas object detection only captures coarse-grained spatial information through bounding boxes. The lack of detailed spatial information contributes to the lower dataset classification accuracy on bounding boxes (61.5%) compared to segmentation masks (67.5%).

• Caption: Caption discards all spatial information, creating more discriminative semantic representations through natural language. This textual representation is less affected by the low-level and spatial cues in the images.

• Edge detection and SAM contour: Object contour delineates fine-grained object shape and spatial information, while lacking the rich object semantic information present in semantic segmentation masks and bounding boxes.

We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] Thomee et al, YFCC100M: The New Data in Multimedia Research
[2] Gadre et al, DataComp: In search of the next generation of multimodal datasets
[3] Changpinyo et al, Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
[4] Li et al, Exploring Plain Vision Transformer Backbones for Object Detection
[5] Kirillov et al, Segment Anything
[6] Podell et al, SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
[7] Anthropic, Claude 3.5-Sonnet
[8] Meta, Llama 3.1

My concerns are mostly addressed, it would be interesting to see how these suggestions could be incorporated to build a new dataset curation method in future work. I increase my score to 6 after reading the rebuttal.

Many thanks for your valuable feedback and discussion!

We thank the reviewer for the review and the insightful questions. We would like to address your concerns below.

w1: Could it be possible that a combination of multiple visual/semantic attributes jointly contributes to the large overall bias? This needs to be verified.

Thank you for this suggestion. To verify this, we consider different combinations of 3 transformations with lower dataset classification accuracies: object detection (61.5%), random-order pixel shuffle (52.2%), and SAM contour (73.1%). For each pair of transformations, we combine them by concatenating the resulting transformed images on the channel dimension.

We observed that combining semantic and structural attributes resulted in higher dataset classification accuracy compared to using a single attribute alone.

w2: So if we use the smaller ResNet-50 model, can we still observe the high accuracy over each individual structural or semantic attribute?

We are happy to provide more results on this. We’ve expanded our main experiments to employ smaller vision architectures: ResNet-50 (23M parameters) [1] and ConvNeXT-femto (5M parameters) [2]:

Additionally, we use two weaker sentence embedding models (MPNet-Base [3] and Sentence-BERT-Base [4]) for caption classification:

Across model sizes and architectures, the classification accuracy over each individual structural or semantic attribute remains high. We’ve added this result to our Appendix.

w3: So when we are attempting to create a debiased dataset in future, how can we make sure our newly collected data are truly diversified and fair? Can the authors provide similar evidence over 'Memorization vs. Generalization' as in [1] Sec. 4.2., and verify individual inter-dataset bias attributes with a truly unbiased pseudo-dataset?

We agree that our framework can only identify bias relative to the given combination of datasets (e.g., YCD) rather than their bias to the universal vision world. Nevertheless, identifying inter-dataset bias can still be helpful in creating a debiased dataset, which is 'truly diversified and fair', in the future. For example, in the paper, we used LLM to summarize specific characteristics of each dataset. These characteristics can be used retrospectively to analyze the dataset curation procedure. On YCD:

• YFCC contains predominantly outdoor scenes and human interactions.

YFCC samples images solely from Flickr, a platform for user-uploaded photos. As Flickr users primarily share personal photos, landscapes, and social interactions, YFCC images predominantly feature natural scenes and human activities. Moreover, YFCC excludes photos labeled as “screenshots” [5], reinforcing the focus on human-related and natural imagery.

• DataComp has the lowest number of unique objects per image.

DataComp filters for images with high embedding similarity to ImageNet training examples [6], most of which feature object-centric images. While this empirically leads to higher zero-shot performance of downstream CLIP models, it biases the dataset toward images with lower per-image object diversity.

• CC and DataComp are significantly brighter and contain more digital graphics and object showcase.

These datasets are collected from the Internet and feature results from search engines [6, 7], which prioritize professionally created content like advertisements, infographics, and digital media. This results in a higher prevalence of digital graphics and brighter images, optimized for visual engagement and online presentation.

We observe similar trends over 'Memorization vs. Generalization' as in Sec. 4.2 of [8] on our transformed datasets. Specifically, we create 3 pseudo-datasets, all of which are sampled without replacement from the same transformed YFCC dataset (e.g., images of Canny-detected edges or images of object bounding boxes). We perform dataset classification on these pseudo-datasets. The tables below present the pseudo-dataset classification training accuracy on YFCC bounding boxes and YFCC Canny-edges. As the number of training images from each dataset increases, the task becomes harder. All of the models trained on this pseudo-dataset classification have a chance-level accuracy of 33% in the validation set. This is because they merely memorize the dataset origin of each training image rather than learning any generalizable patterns.

q1: Another slight concern is about the presentation. Since an ideal debiased dataset should enjoy an equal likelihood to be predicted as one of the anchor datasets (e.g. YCD), maybe the authors should explicitly clarify this important task setting as early as possible in the paper.

We appreciate your suggestion. We’ve added a clarification that “An ideal unbiased dataset should have a chance-level probability of being predicted as any of the anchor datasets.” in our introduction.

We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] He et al, Deep Residual Learning for Image Recognition
[2] Liu et al, A ConvNet for the 2020s
[3] Song et al, MPNet: Masked and Permuted Pre-training for Language Understanding
[4] Reimers & Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
[5] Thomee et al, YFCC100M: The New Data in Multimedia Research
[6] Gadre et al, DataComp: In search of the next generation of multimodal datasets
[7] Changpinyo et al, Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
[8] Liu et al, A Decade’s Battle on Dataset Bias: Are We There Yet?

I greatly appreciate the authors effort in putting up the response. In fact, it's beyond my expectation. I find my 3 major concerns have been adequately addressed thanks to the additional information - I trust the authors will incorporate them into the final version.

In retrospect, I believe what hindered my initial impression was the title of the paper, which led me to believe this was some incremental work over investigating 'intra-dataset biases', such as imbalanced distribution of object classes or visual features and etc. I highly recommend, if applicable, that the authors explicitly open up with 'INTER-dataset biases' in the title or somewhere early-on in the revised paper. Overall, I am willing to update my rating owning to this thorough rebuttal.

Thank you again for your helpful comments and for reviewing our response! We are glad to hear that the concerns have been addressed. We will incorporate the additional results discussed in our rebuttal into the next version of the paper and emphasize that we're studying the inter-dataset bias in our introduction to better reflect the focus of our work.

We sincerely thank you for your constructive comment. We are encouraged that you find our paper provides helpful insights about bias in large-scale datasets. We address your concerns below:

w1: How should one interpret these results under the argument that neural networks in general can learn noisy labels, and can fit arbitrary distributions? How does the baseline 81.7% performance change if you assign random labels to the original set of 3M images?

All the reported accuracies in our work are evaluated on 30k validation samples. On the other hand, the findings in [1] about high model accuracy on random labels through memorization are only on the training set. When we assign random labels to our data, the model only achieves a chance-level accuracy of 33.3% on the validation set. This is expected since models trained on random labels cannot learn dataset-specific patterns that is generalizable to validation sets.

w2: What happens if you train directly on the semantic masks itself?

We would like to clarify that we converted each pixel, rather than the full image, to a 150-channel binary array (Line 79). The transformed image maintains the original width and height.

Additionally, training on the RGB segmentation masks with 2 different color palettes results in stable accuracies of 70.1% and 70.0%.

w3: For the caption classification task (sec 3.2), could you also use another sentence embedding model that is potentially weaker in its initial MTEB performance?

We used weaker models MPNet-Base [2] and Sentence-BERT-Base [3] for caption classification:

The accuracy is consistent on all 3 sentence embedding models.

w4: Could you sample an entirely different set of 3M images and 30K validation images, rerun the experiments, and cross-validate on the two different validation sets?

We reran the experiments on new samples (3M training, 30K validation). Due to time constraints, caption and segmentation experiments are on 300k new training samples. Accuracies vary minimally across two samples.

w5: For Fig 11, could you redo that analysis on another subset of 3M samples as well? … Can the authors provide a reconciliation between this prior result and their results?

We observe the number of overlapped top 10 object categories (as in Fig 11) is high between the original 3M and new 3M images:

This does not contradict [4]. Fig 11 shows the top 10 object classes with the highest proportion of their images from a particular dataset, rather than the most frequent object classes within each dataset. Despite certain object categories being overrepresented in certain datasets, the overall object distribution vectors [4] are highly correlated:

w6: Another interesting analysis ... would be to use the VisDiff tool to perform an analysis on the visual distribution differences between the pairs of YCD datasets.

Thank you for recommending VisDiff [5]. We generated caption-based set differences for each dataset pair (Figure 2 in the attached pdf):

Results from VisDiff highly overlap with LDA- and LLM-extracted dataset characteristics, emphasizing “people” and “outdoor activities” for YFCC and “product” for DataComp. We’ve added this to our Appendix.

w7: There are no concrete suggestions/discussion on how one could go about mitigating these biases.

Our study provides a general framework for identifying the concrete form of low-level and semantic bias in large-scale datasets. The identified dataset bias can be used retrospectively to analyze the dataset curation procedure. For example, on YCD:

• YFCC contains predominantly outdoor scenes and human interactions.

YFCC samples images solely from Flickr, a platform for user-uploaded photos. As Flickr users primarily share personal photos, landscapes, and social interactions, YFCC images predominantly feature natural scenes and human activities. Moreover, YFCC excludes photos labeled as “screenshots” [6], reinforcing the focus on human-related and natural imagery.

• DataComp has the lowest number of unique objects per image.

DataComp filters for images with high embedding similarity to ImageNet training examples [7], most of which feature object-centric images. This biases the dataset toward images with lower per-image object diversity.

• CC and DataComp are significantly brighter and contain more digital graphics and object showcase.

These datasets are collected from the Internet and feature results from search engines, which prioritize professionally created content like advertisements, infographics, and digital media. This results in a higher prevalence of digital graphics and brighter images, optimized for visual engagement and online presentation.

We have consolidated the above as a discussion paragraph in our draft.

We thank you again for your valuable feedback and we hope our response can address your questions. If you have any further questions or concerns, we are very happy to answer.

[1] Zhang et al, Understanding deep learning requires rethinking generalization
[2] Song et al, MPNet: Masked and Permuted Pre-training for Language Understanding
[3] Reimers & Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
[4] Udandarao et al, No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
[5] Dunlap et al, Describing Differences in Image Sets with Natural Language
[6] Thomee et al, YFCC100M: The New Data in Multimedia Research
[7] Gadre et al, DataComp: In search of the next generation of multimodal datasets
[8] Changpinyo et al, Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

I thank the authors for their added experimental results and comments.

(1) The random labels experiment sufficiently answers my question regarding the specificity of these dataset-level biases.

(2) Thanks for running the sentence embeddings and segmentation mask experiments, those high stable accuracies clarify my concern.

(3) The cross-validation results are very strong. I would encourage the authors to somehow incorporate these results into the main paper perhaps as a mean +/- std or something similar, I think this would significantly increase the confidence in the paper's findings.

(4) Thanks for the clarifying discussion b/w your work and the No Zero-shot paper. You being able to reproduce the findings of that paper are very interesting.

(5) Great that the VisDiff results corroborate your analysis.

(6) Your response to w7 is very insightful, thank you for adding this. I would encourage to expand on this discussion and if possible add some sample images from each of the datasets to that section in the appendix. Would be super useful!

Overall, the author rebuttal has significantly increased my confidence in the paper and the main results and insights provided in the paper are very useful. I am updating my score to an 8.

Thank you for your time and feedback on the paper! We’re glad that your concerns have been resolved. We will make sure to include the additional results from the rebuttal in the next version of our paper. The review has been very helpful in enhancing our paper.