Will the Inclusion of Generated Data Amplify Bias Across Generations in Future Image Classification Models?

Questions:

[1] The experiments: If possible, I would like to see the results from other standard datasets or more subgroups for the dataset used.

A: As you suggested, we conducted experiments on the Living-17 dataset, which consists of 17 superclasses, each containing 4 subclasses. We utilized the Stable Diffusion 1.5 model as the base generative framework to study the inheritance phenomenon of bias across generations. The results are presented in Appendix A.

[2] Citations: Some original works in the introduction section and later are missing citations. If the authors think they have cited all the necessary work, please put that in a rebuttal.

A: Thanks for your suggestion. We have cited more work about using generated data helping the model traiing in our revision. It would be appreciate if you could suggest some work for further discussion.

[3] Conclusion and future work: I think it can be improved or extended to connect with the problem and story.

A: Thank you for your suggestion. Indeed, there are several models that involve self-consumption loops, such as Stable Diffusion, LLaMA, LLaVA, and Nemotron. Notably, Nemotron is trained on a dataset consisting of more than 98% synthetic data. While the inclusion of synthetic data in the training process has shown some benefits, it is important to consider the potential harms, especially concerning model biases. This concern has prompted us to investigate the impact of generated data on model performance and bias, particularly as the number of self-consumption loops increases.

We plan to expand on this in our future work by further exploring how the inclusion of synthetic data in iterative training cycles influences not only model performance but also the ethical implications of model biases. This line of inquiry could provide valuable insights for future AI model development.

Weaknesses:

[1] The paper mostly conducts experiments on three datasets. Among the three datasets, Colourized-MNIST and CIFAR20/100 are very small datasets in terms of the resolutions and the number of data and classes compared to the existing image data. Moreover, the largest model used in the paper is ResNet50, which is a relatively small network when compared to SOTA models like ViT. This raises the concern of whether the observations from the experiments are still valid in realistic scenarios with large datasets and models.

A: On small datasets, CNN-based models are more suitable as ViT-based models tend to suffer from severe overfitting issues without large-scale datasets. For this reason, our primary analysis focuses on CNN-based models.

As you suggested, we conducted additional experiments on the Breeds dataset, which is organized based on subgroup connections identified by WordNet. Additionally, we evaluated the performance of ViT models. The results are presented in Appendix A.

[2] Some experimental results are hard to understand.

• baseline performance. To my understanding, CIFAR20/100 has a smaller number of classes and should be an easier dataset to classify when compared to CIFAR100. However, the baseline performance of ResNet50 on CIFAR100 is around 80%; on CIFAR20/100 is around 50%.
• The trends are inconsistent between models. For example, in Fig. 6(a), ResNet50 and LeNet show an opposite trend in Equal Opportunity and Disparate Impact but the same trend in average accuracy and Maximal Disparity; VGG-19 remains unchanged for the tested metrics.
• The trends are inconsistent between datasets. For example, comparing the ResNet50 baseline between Fig 5(a) on CIFAR-20/100 and 6(a) on CIFAR-100. Even though the datasets are similar, the ResNet50 baseline shows a totally different trend for the Equal Opportunity, Disparate Impact, and Maximal Disparity metrics.
• Similar discrepancies are noted in other sections of the paper. The experimental findings do not appear to explain how the generated data affects model bias in general.

A:

• Baseline performance:
  The CIFAR20/100 dataset consists of 20 superclasses, each containing 5 subclasses. Due to its low resolution and significant distribution disparity between subclasses within a superclass, it is challenging for models to learn generalized representations from this dataset. This results in lower classification accuracy compared to the original CIFAR100 dataset. Similar experimental results can be found in [a].

• Inconsistent trends between models:
  The impact of generated data varies across models due to differences in architecture and model capacity. On one hand, generated data can enhance model training and improve performance. On the other hand, inherent explicit biases in generated data may harm model performance. Learning from data augmented with generated samples requires balancing these two effects. Variations in model architecture and capacity result in inconsistent trends across models and datasets.

• Inconsistent trends between datasets:
  The trends differ between datasets like CIFAR20/100 and CIFAR100 due to differences in data distribution and subclass structures. These disparities influence how models learn and respond to generated data, leading to different observed trends.

• Explanation of experimental findings:
  Detailed explanations of the experimental findings are provided at the end of each experiment and in Section 5.

[a] Zhang et al. "Discover and Mitigate Multiple Biased Subgroups in Image Classifiers." CVPR 2024.

[3] The conclusion is weak. Based on the observations, the authors conjectured that the model bias is affected by multiple factors, including the datasets, models, and data quality across generations. However, the authors did not provide clear experiment evidence or solid theory explaining how these factors influence the model bias.

A: This work is an empirical study aimed at understanding the impact of generated data on the bias of image classification models during the self-consuming loop. Our conjectures are based on experimental observations and serve as a preliminary explanation of the results. We hope our research will inspire future studies to investigate the underlying mechanisms in more depth. To facilitate further research, we will open-source all the code and data used in this project.

Weaknesses: [1] The paper presents mixed findings across datasets and models but does not provide in-depth explanations for these variations. While section 5 includes some discussion on the root causes of observed behaviors, this analysis remains at a high-level and is not well-supported or directly connected to the experiments in earlier sections. The analysis would be more convincing with clearer connections to the results, reinforcing the paper’s claims with evidence from the experiments.

A: Thank you for your suggestion. This work serves as an empirical study to translate real-world practices into a case study, investigating whether generated data can influence model bias across generations.

We include result analysis after each experiment and observe that there is no consistent pattern across different models and datasets. For this, we propose a hypothesis in the final section, considering multiple factors that may contribute to these findings. We hope our work inspires future research to delve deeper into this phenomenon.

[2] In table 1, FID fluctuates for Color-MNIST and CIFAR10 after several rounds of data generation, while it increases substantially for HARD-ImageNet starting from the second iteration. This trend suggests a marked difference in data quality for HARD-ImageNet compared to the other datasets. However, the subsequent experiments focus primarily on how generated data impacts downstream performance and bias without addressing how this observed FID trend might influence these results. A discussion on how does the data quality(assesed by FID) could affect interpretations across the three datasets would enhance the clarity of the findings.

A: It is important to note that the generated data in the self-consuming loop not only influences the image classification model but also affects the generative model itself.

For MNIST and CIFAR, we employ a GAN model trained from scratch. In this scenario, the consistent inclusion of generated data does not drastically affect the generative model, as clean data still constitutes the majority of the training dataset. This explains why the FID scores for MNIST and CIFAR remain relatively stable across generations.

In contrast, for Hard ImageNet, we use the pre-trained Stable Diffusion model, where both the clean data and the generated data across generations differ significantly from the pre-training dataset. As the number of generations increases, these distribution differences accumulate, resulting in a substantial increase in the FID score. This phenomenon underscores the sensitivity of pre-trained generative models to distribution shifts in a self-consuming loop.

The impact of generative data on model bias across generations depends on several factors, including the generative model's performance and the learning capacity of downstream models. While the generative model may produce more data for certain subgroups, the quality of the data—reflected in the FID score—ultimately determines whether the performance of these subgroups

[3]Some methodology details are lacking, making it challenging to fully understand and replicate the study. For example, in section 3.2, there is limited information on the design of the human study, the impact of expert-guided filtering on image quality, and the specific r% used. The paper would benefit from some recommendation on the usage of generated data in generative models or downstream tasks with the insights from the experiments.

A: We will open-source all code, models, and data to facilitate future research. The data filtering process is detailed in the questions section. In this study, we observe that the ratio of generated data used for data augmentation must be carefully considered to mitigate model crashes caused by distribution disparities between generated and real-world data. Additionally, it is worth noting that for a given model and dataset, the trend in model bias changes is usually consistent. This consistency suggests that we can adjust the mixup ratio by observing a small number of iterations and dynamically adapting our augmentation strategy.

Weaknesses: W1. Lack of experiments on the choice of generative models. Various generative models can differ in behavior, the choice of model likely impacts sample quality and influences the outcomes of subsequent studies.

A: Thank you for your suggestion. We agree with your observation that different generative models can have varying impacts on downstream tasks.

Criteria for Selecting Generative Models In our study, we utilized three datasets: MNIST, CIFAR, and an ImageNet-like dataset. MNIST and CIFAR are small-scale datasets with low resolution, which led us to select GAN models for learning their distributions, as diffusion models often struggle with limited-size data. In contrast, the ImageNet-like dataset, with its higher resolution, is more suitable for diffusion models.

We also considered the point you raised in our study. However, due to limited computational resources, we conducted experiments on each dataset using only one type of generative model.

Our study aims to shed light on the issue of continuously using generated data in real-world applications, serving as a case study. We hope this work will inspire further investigations into how various generative models impact bias in downstream tasks.

W2. The motivation of the paper is on future image classification synthetic data plays in it. With foundational models playing a dominant role, integrating settings like synthetic data for transfer learning in classification will enhance the paper better, going beyond the current base case that may lack scalability.

A: Thanks for your suggestion. We will include this discussion into our paper.

W3. The experiments are mainly targeting the model bias within a self-consuming loop in image classification domain. However, the conclusions/observations are not significant.

A: We acknowledge that there is no universal rule for the impact of generated data within a self-consuming loop on downstream image classification models. However, it is worth noting that the observed trends for each model on the same dataset remain consistent, providing guidance for real-world model development practices.

This indicates that while it is known to us from this study that using generated data can influence model bias across generations, we can simplify the process by training models over a few generations and observing the trend. If the bias of interest continues to worsen across the observed generations, this suggests the need to incorporate additional real-world samples to mitigate the adverse effects of generated data on the model.

Questions:

Q1. How many synthetic data are added or ratio p? Is there a rationale behind the choice of p.

A: We maintain the size of the generated data at 10% of the original clean dataset. Previous studies have highlighted that the ratio between synthetic data and clean data should be carefully considered. A large ratio can lead to model collapse, while a small ratio is expected to enhance performance as desired.

Q2. Are there any experiments or preliminary results on tasks with a larger number of classes?

A: We further conduct experiments on the Breeds dataset (sub ImageNet dataset)[a], which is organized by the subgroup connections established in WordNet. The results are shown in appendix. A.

[a] Santurkar et al. "Breeds: Benchmarks for subpopulation shift." ICLR 2021.

Q3. How is CLIP used in filtering? Specifically, is it based on the similarity score between label texts and images?

A: We use the similarity score betwen the label text and images.

Q4. Would different losses lead to varying results?

A: In our study, we work on the task of image classification, where the cross-entropy is the most popular metric for learning. In our view, the change of loss would not lead to varying results. Because The bias comes from the distribution of generated data, originally from the imbalanced generation of the generative model. Thanks.

Q5. Any insights on how could the conclusion generalize to other tasks, or other modality?

A: Model bias originates from the data it learns. Due to the imbalanced data generation by generative models, the distribution of augmented data often exhibits unevenness, introducing bias into downstream tasks. This phenomenon not only affects classification tasks, as studied in our work, but also extends to other domains such as robotic control, visual question answering, and more.

Thank you for your response and suggestions! We completely agree that adding experiments on fine-grained settings (e.g., loss functions, downstream tasks) would further enrich the paper.

However, within the limited scope of a conference paper, we have chosen to focus on a more general setting on the most widely used task for concept verification: the standard image classification task on standard image datasets using the cross-entropy. Specifically, we employ popular generative models (GAN and Stable Diffusion) to enhance the training process during the self-consuming loop. This approach allows us to examine the impact of generated data on model performance in general, aligning closely with the central theme of our paper's title.

It is undoubtedly valuable to explore scalability across other downstream tasks, such as transfer learning, domain adaptation, and the impact on model robustness. We leave this broader investigation to future work, aiming to inspire more researchers to bridge the gap between industry practices and academic research while fostering mutual advancements.

We hope this addresses your concern further.