Are Synthetic Classifiers Really as Good as Real Classifiers?

Thanks for your detailed reading and pointing out problems of our work. We have revised the paper carefully following your suggestions. Below, we address your main concerns on the paper content.

Q1: Extending the experiments to non-visual modalities or providing some preliminary analysis could have increased the paper’s broader applicability.

A1: Thank you for pointing that out. Indeed, this paper focuses mainly on vision tasks, but understanding the effectiveness of a small amount of real data for synthetic data representation learning can still provide valuable insights that may be applicable across various domains. Here, we investigated the impact of RealTune on GPT2 trained on synthetic data generated by GPT4. The real data and synthetic data for pretraining both consist of 0.11M texts, and the finetune data size is 20% of the pretraining data. The models are pretrained for 15k steps and finetuned for 1k steps.

The results are shown in the table below. We observe that RealTune lead to a decrease in loss by 0.7 for GPT2-Syn, narrowing the gap with GPT2-Real, while SynTune resulted in a loss increase of 0.22 for GPT2-Syn. This indicates that RealTune is effective for text tasks as well, demonstrating its generalization.

Experimental results are added in Section 4.4.

GPT2 loss of different finetune methods. GPT2-Real represents a model pretrained on real data, and GPT2-Syn represents a model pretrained on an equivalent amount of synthetic data.

Q2: The analysis lacks an adequate diversity of datasets that could highlight other real-world challenges. ImageNet-X, though useful, does not fully encompass the variety of rare scenarios that might be seen in real-world applications. For a thorough evaluation, incorporating other benchmarks like ImageNet-C or ObjectNet could have provided a more comprehensive assessment of classifier robustness.

A2: Considering evaluation on ImageNet-C is reasonable and valuable. In fact, paper [1] has compared the performance of the synthetic and real classifiers (ViT and CLIP) on ImageNet-C and ImageNet-3DCC (which includes 12 common corruptions that take depth into account, e.g., z-axis blur, far and near focus errors, etc.). Their results show that the performance of synthetic classifiers on the two datasets is significantly lower than that of real classifiers, highlighting the limitations of synthetic data. This has been mentioned in the related work of our paper.

Through evaluation on ImageNet-X, we identified challenges for synthetic classifiers that are more relevant to real-world scenarios and model semantics. We added the discussion regarding ImageNet-C from paper[1] into Section 2.1.

Reference:

[1] Krishnakant Singh, Thanush Navaratnam, Jannik Holmer, Simone Schaub-Meyer, and Stefan Roth. Is synthetic data all we need? benchmarking the robustness of models trained with synthetic images. In CVPR, 2024.

Q3: Some figures in the paper are difficult to interpret due to suboptimal presentation choices.

A3: Thank you for your suggestions. We made slight adjustments to Figure 4(a) and added a more detailed caption. Additionally, we replaced Figure 4(b) from a predicted label histogram to a fine-grained confusion matrix of 8 snake species for a more intuitive understanding of fine-grained class confusion in the revision.

Q3：There is no comparison conducted w.r.t prior arts that work with training models using mixture of synthetic data and real data. As surveyed by the authors themselves in the related work section, yet the authors compared with none of them.

A3: Thanks you for reminding us. In the revision, we added a comparison with the method described in paper [1] which mentioned in our related work section. For clarity, we refer to our method as "MixData" and the method from paper [1] as "MixLoss." In "MixLoss," the losses of real data and synthetic data are summed at each iteration for backpropagation while our "MixData" combines real and synthetic data for training using the standard forward and backward methods. As shown in the revision (quoted below), accuracy of MixData pretraining higher than that of MixLoss. After RealTune, the model performance of MixData further improves, while the model performance of MixLoss decreases. Therefore, our MixData outperforms MixLoss. The results are added in Appendix B.

For other related work [2-7], we observe that they are specifically tailored towards particular directions or models, making them unsuitable for direct comparison with our approach. For instance, [2] is tailored for contrastive learning by reducing the augmentation strength for generating positive and negative samples, [3] for semantic segmentation, [4] for optical flow, [5] for the medical field, [6] for CLIP pretraining, and [7] primarily for traffic object detection. Nevertheless, investigations across diverse domains [2-7] also underscore the significance of exploring the mixing of real and synthetic data.

Comparison of different mixing methods. MixData is our method, and MixLoss is from paper [1]. MixData+RealTune (MixLoss+RealTune) refers to training the model with MixData (MixLoss) and then fine-tuning it with real data.

References [1] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XI- AOJUAN QI. Is synthetic data from generative models ready for image recognition? In ICLR, 2023.

[2] Yifei Wang, Jizhe Zhang, and Yisen Wang. Do generated data always help contrastive learning? In ICLR, 2024.

[3] Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa. Learning from synthetic data: Addressing domain shift for semantic segmentation. In CVPR, 2018.|

[4] Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T Freeman, and Ce Liu. Autoflow: Learning a better training set for optical flow. In CVPR, 2021.

[5] Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification. Neurocomputing, 321:321–331, 2018

[6] Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In CVPR, 2024.

[7] Viktor Seib, Benjamin Lange, and Stefan Wirtz. Mixing real and synthetic data to enhance neural network training–a review of current approaches. arXiv preprint arXiv:2007.08781, 2020

Q4：Training models with both synthetic data and real data is simply not novel at all.

A4: Using both synthetic data and real data for model training has indeed been explored in many studies. However, the design for using both is not the primary focus of our paper. What we aim to investigate is whether real data can bridge the significant gap between synthetic and real classifiers, and at what cost this gap can be bridged. Our research in Section 4 reveals that by using a very small amount of real data for a short finetuning period, we can rapidly bridge this gap, a finding that has not been revealed in previous articles.

Thank you again for your careful reading. We have carefully refined our paper following your suggestions, and addressed each of your concerns above. We respectfully suggest that you could re-evaluate our work based on the updated results. We are very happy to address your remaining concerns on our work.

Thank you for your careful reading and pointing out our problems! We have carefully revised the paper following your suggestions. We address your concerns as follows.

Q1: The claim that synthetic classifiers perform worse on fine-grained dataset and rare scenarios is not well justified.

(1) From Tab. 1 and Fig. 2, synthetic classifiers and real classifiers show similar trend on both IN results and fine-grained results. Synthetic 371M shows very similar accuracy as real 64M.

(2) In Fig. 3, the so called relative accuracy difference is 0.5%.

A1：

From Tab. 1 and Fig. 2, synthetic classifiers and real classifiers show similar trend on both IN results and fine-grained results. Synthetic 371M shows very similar accuracy as real 64M.

(1) As you have mentioned, it is evident that synthetic classifiers and real classifiers exhibit a similar trend in both IN results and fine-grained results. This similar trends highlights that the inferior performance of synthetic classifiers (IN results) can be mainly attributed to their tendency to confuse closely related subclasses (lower fine-grained accuracy).

For CLIP-Real-64M v.s. CLIP-Syn-371M, the checkpoints we used from paper[1]. And paper[1] did not report the models' variance or provide multiple checkpoints for us to calculate variance. And it was challenging for us to re-pretrain CLIP due to the computational demands of generating synthetic data and pretraining. However, it is generally believed that the variance at the ImageNet level is within 0.3. Therefore, we consider the differences of 83.5 vs. 83.1 (CLIP-Real-64M v.s. CLIP-Syn-371M) to be significant. Additionally, at an equal data size, the gap of fine-grained results between real and synthetic classifiers are more significant.

References:

[1] Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In CVPR, 2024.

In Fig. 3, the so called relative accuracy difference is 0.5%.

(2) Regarding the statement "In Fig. 3, the so-called relative accuracy difference is 0.5%," we find we made an error in labeling the y-axis scale during plotting Fig. 3. It should have been 50% instead of 0.5%. From the modified Fig. 3, we can see that synthetic classifiers perform significantly worse than real classifiers in certain rare scenarios. Taking CLIP at equal data size as an example, CLIP-Syn-371M performed 15% lower on darker images, 20% lower on smaller objects, and 40% lower on person blocking compared to CLIP-Real-371M as shown in Figure 3. Thank you for pointing out this important mistake. We have corrected Fig. 3 in the revision.

Q2: The fact that generative models fail to generate fine grained is simply not true. There are many works that focus on generating images in the context of fine-grained images. For instance, SphericGAN: Semi-supervised Hyper-spherical Generative Adversarial Networks for Fine-grained Image Synthesis; Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation; The authors should study the synthetic classifiers with data generated with models designed specifically for Fine-Grained Images.

A2: Regarding the potential for better models or specialized methods to address the issue of fine-grained classes is a very nice suggestion. In fact, in our paper, we also attempted various approaches to enhance the generation capabilities of fine-grained classes, such as adjusting classifier-free guidance, modifying prompts, and switching to better generative models. However, as shown in Figure 4a, increasing the CFG scale can improve class consistency but still falls short of closing the gap with real data. And increasing CFG will lead to synthetic datasets lack of diversity, resulting in a decrease in the synthetic classifier's performance. Therefore, this method-specific approach does not entirely resolve the issue, as while it may improve this particular problem, it could result in a decline in other aspects.

Regarding the two papers you mentioned, as they do not provide code, it is challenging for us to reproduce these experiments within the limited time. If you have access to the code, we would be happy to compare our results with theirs. Anyway, we believe that, just as we have attempted in Figure 4a, if they may excel in fine-grained aspect, they might fall short in other aspects (e.g., diversity).

Q6: On a related note, it might helpful context to report how other pretrained models fare when finetuned on a small random subset of ImageNet. I am curious how other modern self-supervised pretraining strategies might do for example.

A6: Thank you for your insightful recommendation! Due to the extensive time requirements of conducting experiments on ImageNet, we conduct experiments on DINO using ImageNet100. The random subset used for RealTune consists of randomly selecting 100 images per class in ImageNet100 (~7.7% of ImageNet100). The results are as follows. The results indicate that without RealTune, DINO-Syn exhibits an accuracy 32.72% lower than DINO-Real, which is a significant gap. However, after RealTune was applied to DINO-Syn, the performance gap between DINO-Syn and DINO-Real narrows to 7.62% (DINO-Real Vanilla v.s. DINO-Syn RealTune acc), further highlighting the effectiveness of RealTune in self-supervised learning. The results are added to Appendix B.

ImageNet100 classification accuracy of different finetune methods on DINO. DINO-Real represents pretraining on Real ImageNet100, while DINO-Syn represents pretraining on synthetic ImageNet100.

Thank you again for your careful reading. We have carefully refined our paper following your suggestions, and addressed each of your concerns above. We respectfully suggest that you could re-evaluate our work based on the updated results. We are very happy to address your remaining concerns on our work.

Q4: The citation provided discussing whether real world data is limited (Villalobos et al 2024) makes a comment about there only being a couple trillion images taken each year while in this investigation we are looking at the impact of tens of thousands of images. How much are we at risk of running out of data?

A4: The issue of running out of data is indeed a controversial topic. From our perspective, although humans generate vast amounts of real data, there is not much high-quality data. In contrast, generative models, with the ability to produce data in specified directions under our control, may possess higher quality. This is why we compare the differences between real and synthetic data in model training.

Since the process of generating data is time-consuming and resource-intensive, we conducted the experiment in a more controllable setting. This is why our experiments’ scale do not reach the trillion-level. To ensure fair comparisons, we examined two settings: (1) equal data size and (2) equal accuracy, as quantitative experiments on real-world scenarios. The results indicate that due to data quality issues, there are still discrepancies when using synthetic data for model training. However, these discrepancies can be quickly bridged with a small amount of real data. Therefore, designing appropriate training strategies can enhance the effectiveness of synthetic data.

Q5: The real question someone might want answered is how to get a strong classifier with large numbers of unlabeled/weakly-labeled images and a small amount of real labeled data. I am not sure the most convincing case is being made that is worth it go through the roundabout process of producing millions of outputs from a generative model while mixing in a little bit of real data. How does performance compare to sampling and training on say, a million LAION images whose captions match up to the imagenet labels? Would that be a reasonable point of comparison? Similarly how would this compare to a model that is pretrained directly on LAION in the style of CLIP?

A5: There are numerous works currently exploring representation learning using synthetic data [1,2,3,4,5]. [1] demonstrates that the accuracy of CLIP trained on synthetic data generated by Stable Diffusion can surpass CLIP trained on real data. [2] shows that when the generative model does not use additional real data, the model trained by mixing synthetic data and real data can also outperform the model trained on real model. What we want to study in our paper is whether synthetic classifiers still have some common defects even if the synthetic model reaches the accuracy of the real model, and whether we can further unleash its potential by identifying and solving the existing problems. The results show that synthetic classifiers do have defects and these defects can be quickly remedied by finetuning a small amount of real data.

Reference:

[1] Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In CVPR, 2024.

[2] Yifei Wang, Jizhe Zhang, and Yisen Wang. Do generated data always help contrastive learning? In ICLR, 2024.

[3] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions on Machine Learning Research, 2023.

[4] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. In NeurIPS, 2023.

[5] Alceu Bissoto, Eduardo Valle, and Sandra Avila. Gan-based data augmentation and anonymization for skin-lesion analysis: A critical review. In CVPR, 2021.

Thanks for your detailed reading and appreciating the exposition and evaluation of our work. We have revised the paper carefully following your suggestions. Below, we address your main concerns on the paper content.

Q1: I don't know that I agree that these failures are in fact specific to models trained on synthetic data. Looking at Figures 3 and 8, all of the error modes are very highly correlated between models trained on either real or synthetic images.

A1: Indeed, the dataset itself (whether real or synthetic data) may inherently contain these failure modes. However, in our paper, we aim to compare the additional errors present in synthetic classifiers compared to real classifiers. Therefore, under the condition where other influencing factors are held constant (i.e., model architecture, training process), we compared two settings: (1) equal data size and (2) equal ImageNet accuracy. The results show that, even with equal accuracy, synthetic classifiers still perform worse in certain rare scenarios compared to real classifiers. For instance, CLIP-Syn-371M exhibited a 10.84% lower performance on larger objects, 5.38% lower on darker images, and 13.65% lower on brighter images compared to CLIP-Real-64M, as illustrated in Figure 3. This indicates that these errors stem from deficiencies in the synthetic data compared to real data.

Additionally, we apologize for the mistake in the y-axis scale of Figure 3 and Figure 8. The y-axis scale should have been enlarged by a factor of 100. This has been corrected in the revised version. From the modified Fig. 3, it is evident that synthetic classifiers perform significantly worse than real classifiers in certain rare scenarios.

Q2：One issue I have is that the authors do not discuss how much data goes into training the image generator. … So do the billions of images used to train StableDiffusion not count when discussing how much data is needed to get these results?

A2: This is a very good question! The amount of data used to train a generative model is indeed an important additional dimension. In the comparison and analysis, we only consider the amount of data required to train classifiers, that is, we only consider the amount of synthetic data used to train the synthetic classifier followed [1], which ensures a fair comparison with the amount of data required to train real classifiers. Since Stable Diffusion is a public open source model, training Stable Diffusion is not our cost when training synthetic classifeirs. Additionally, if we were to consider the data used to train the generative model, it would imply that synthetic classifiers, which consumes a large amount of data, still performs worse than real classifiers. This further validates the shortcomings of synthetic data in model training.

Reference:

[1] Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In CVPR, 2024.

Q3: I find it interesting that some of the specific failure modes seem easily addressed by data augmentation. There is focus in Section 3 on how to prompt the image generator to produce smaller/larger objects or whether other generators don't have the same light/dark limitations, but cropping and resizing and brightness augmentations could all trivially expose the model to more diverse object sizes and image conditions, no?

While maybe somewhat orthogonal, were any tests of data augmentations done to address any of the effects discussed here?

A3: Whether the failure modes can be solved through augmentation is a question that is worth studying. We conducted a more detailed ablation study on augmentation. We compare models without augmentation (vanilla), brightness adjustment augmentation, blocking augmentation, crop augmentation, and models with a combination of brightness adjustment, blocking, and cropping. The results are as follows. The performance gap between models with and without augmentation is minimal, and the difference between SynTune and RealTune after using augmentation has not decreased. This indicates that augmentation cannot easily addressed synthetic data failure modes. What matters is RealTune, underscoring the importance of real data.

The results are added to Appendix B.

CLIP-Syn-371M ImageNet accuracy under different augmentation methods

We appreciate your careful reading and pointing out the problems. We have diligently revised our paper in accordance with your suggestions. Below we address your main concerns.

Q1: The described scenarios are rather limited and narrow and lack of generalization to broader concepts, e.g., scale, occlusion with human are very specific settings.

A1: We followed the ImageNet-X[1] to identify common defects in synthetic vision models in real-world scenarios. These specific typical issues reflect gap between synthetic data and real data in model training. This analysis can be extended to other domains such as text tasks.

We compared the performance of GPT2 models trained using an equal amount (0.11 million) of real texts and synthetic texts generated by GPT4. The results are presented in the table below. The results indicate that similar to the results in vision tasks, there is a significant performance gap between GPT2-Syn and GPT2-Real, further confirming the limitations of synthetic data in model training. Additionally, we noted that RealTune led to a 0.7 decrease in loss for GPT2-Syn, reducing the disparity with GPT2-Real, while SynTune resulted in a 0.22 increase in loss for GPT2-Syn. This highlights the effectiveness of RealTune for text tasks as well.

Experimental results are added in Appendix B.

GPT2 loss of different finetune methods. GPT2-Real represents a model pretrained on real data, and GPT2-Syn represents a model pretrained on an equivalent amount of synthetic data.

References:

[1] Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nicolas Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, and Mark Ibrahim. Imagenet-x: Understanding model mistakes with factor of variation annotations. In ICLR, 2023.

Q2: Lack of comprehensive evaluation. (1) There is no comparison of baselines with simple data augmentation on object scale and pixel occlusion. (2) Also, Figure 4b mentions class imbalance as an issue for downstream model training. For this specific case, do class rebalancing techniques help solve the issue already

A2：Thank you for your suggestions.

There is no comparison of baselines with simple data augmentation on object scale and pixel occlusion.

(1) To further investigate the impact of augmentation, we conducted a detailed ablation study. We compared models without augmentation (vanilla), brightness adjustment augmentation, blocking augmentation, crop augmentation, and models with a combination of brightness adjustment, blocking, and cropping. The results are as follows. The performance gap between models with and without augmentation is minimal, and the gap between SynTune and RealTune after using augmentation does not decreased. This indicates that augmentation does not address the shortcomings of synthetic classifiers and does not play a significant role here. What matters is RealTune, underscoring the importance of real data. The results are added to Appendix B.

ViT-Syn-1M ImageNet accuracy under different augmentation methods

CLIP-Syn-371M ImageNet accuracy under different augmentation methods

Also, Figure 4b mentions class imbalance as an issue for downstream model training. For this specific case, do class rebalancing techniques help solve the issue already

(2) This seems a misunderstanding of our Figure 4b. We are not suggesting that the synthetic dataset itself exhibits imbalance issues. Instead, we aim to deliver that due to generative models' tendency to confuse fine-grained classes during generation, it leads to imbalanced predicted labels. Essentially, we are still emphasizing that generative models are prone to confusing fine-grained classes. To make it more easily understandable, we replace Figure 4(b) from a predicted label histogram to a fine-grained confusion matrix of 8 snake species for a more intuitive understanding of fine-grained class confusion in the revision.

Q3: In Figure2, the gap between similar ImageNet accuracy models trained with real and synthetic data over the finegrained domain does not appear to be very large - for example. 83.5 of CLIP-REAL-64M vs 83.1 of CLIP-Sync-371M, and 86.21 of ViT-Real-0.25M vs 85.4 of ViT-Sync-2M. It is unclear if these accuracy gaps are statistically significant or within the regime of model sensitivity. Thus it is unclear if the claim of synthetically trained models performing worse on fine-grained categories is technically sound or not.

A3: The ViT and CLIP checkpoints we used from paper[1]. And paper[1] did not report the models' performance variation or provide multiple checkpoints for us to calculate it. And it was challenging for us to re-pretrain ViT and CLIP due to the computational demands of generating synthetic data and pretraining. However, it is generally believed that the variation at the ImageNet level is within 0.3. This is a common practice in ImageNet-scale experiments, and in real world, the variance/stdev is usually much smaller. For example, in ViT paper [2], they report 85.30 ± 0.02 for ViT and 87.54 ± 0.02 for ResNet (Table 2 [2]). Therefore, we believe that a difference by 0.4 (83.5 vs. 83.1) can be considered as a clear difference on ImageNet. Though, we do acknowledge that it is not a very large gap, and the gap is clearer when comparing equal data sizes. We have added this to the footnote to avoid misunderstanding.

Q4: Lack of technical novelty. The idea of pretraining with synthetic data and then fine-tune using a small amount of real data has been adopted in many prior works [1] [2].

A4: Using synthetic data for pretraining and real data for finetuning has indeed been explored in many studies. However, the design for using both is not the primary focus of our paper. What we aim to investigate is whether real data can bridge the significant gap between synthetic and real classifiers, and at what cost this gap can be bridged. Our research in Section 4 reveals that by using a very small amount of real data for a short finetuning period, we can rapidly bridge this gap, a finding that has not been revealed in previous articles.

For papers [1,2], we conduct a detailed comparison of the differences between our method and theirs as follows:

Paper [1]. We note that although they utilize synthetic data to pretrain a model and real data for fine-tuning the model, [1] primarily focuses on using real data and a network (Task2Sim) to select hyperparameters for generating synthetic images most beneficial for downstream tasks. Unlike them, we do not need to modify the hyperparameters of the generative model. Additionally, [1] uses the entire dataset for finetuning, whereas we only require a random subset of a dataset (e.g., 3% of ImageNet). These aspects highlight the efficiency and effectiveness of our approach.

Paper [2]. This is a work closely related to ours. However, we find that in Appendix E of their paper, it is mentioned that "All models are also pretrained on ImageNet." This indicates that their method actually involves three stages: (1) pretraining on ImageNet, (2) pretraining on synthetic data, and (3) finetuning on real data. So this approach is much more cumbersome and time-consuming compared to ours. Additionally, they also use the entire real dataset for finetuning. These comparisons highlight the effectiveness and simplicity of our approach.

[1] Task2Sim: Towards Effective Pre-training and Transfer from Synthetic Data

[2] From Fake to Real: Pretraining on Balanced Synthetic Images to Prevent Bias