IS SYNTHETIC DATA USEFUL FOR TRANSFER LEARNING? AN INVESTIGATION INTO DATA GENERATION, VOLUME, AND UTILIZATION

Q2: Section 4.3 (effect of guidance scale) is of moderate interest, since it may depend on the generative model. There are simple studies that are more general and would give more insights into whether it is worth doing transfer learning with generative models. For example, is it better to train with all "Aircraft" images from LAION-X (possibly without fine-grained labels) and then finetune, or to get fine-grained images from StableDiffusion (different types of aircraft models, which may be hard to query for on LAION-X) and then finetune?

Q3: What are the statistics of the data trained on StableDiffusion with respect to the downstream tasks? For example, for the class Aircraft-Boeing 787 or for a specific car model, it may be the case that LAION-X has more than 3k examples. If one cannot generate more than 3k examples (as in Figure 5, because of computational constraints?), why not use the data directly available? Looking at this and pointing out clear examples where one can generate much more quantity/diversity data than that present in LAION-X would be a good insight. For example, is there a specific type of Aircraft model that is hardly present in LAION-X, but one can generate an infinite amount of realistic images for it?

A2/3: Thanks for your questions. To the best of our knowledge, the reviewer is asking the question of whether using Stable Diffusion data to perform is worth it or not when one can access some real data (for a specific class). To answer this question, we provide two perspectives.

1. Would LAION-X images help for target dataset transfer learning? We do not verify this question since LAION-X does not give accurate labels for each image. Rather, each image is paired with a caption. For example, when we retrieve the images of class “747-200 Airplan” from here (https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fknn.laion.ai&index=laion5B-L-14&useMclip=false&query=747-200+Airplane), lots of different types of airplane have been retrieved in this case. Therefore, it is hard to determine how to adapt LAION-X images into the supervised training pipeline right now. And this is beyond the scope of our paper.

2. To quantitatively convert the effect of synthetic data to the effect of real data, we can study this question in few-shot learning scenarios. We run the vanilla transfer from 1-shot to 50-shot one by one and match the accuracy of the bridged transfer. Therefore, we can measure additional samples by evaluating how much more data the vanilla transfer would need to realize the same accuracy as the bridged transfer. The results for the Aircraft dataset of ResNet-18 are shown below:

It can be found that synthetic data can effectively improve the performance of vanilla transfer and equal to using more images. In reality, collecting the real data may have some other issues like costs and privacy, please see our reply to Reviewer HV3t.

Q4: There may be dataset contamination, which is not discussed, i.e., the test samples of the downstream tasks may be part of LAION-X, while this is not the case for the baseline. Does the model improve performance when prompted with "generate a * with the style of LSUN" without dataset adaptation?

A4: Thanks for your suggestion. We experimented with this prompt and rerun it on the Aircraft dataset, there was no explicit performance improvement with this prompt (within 0.1% accuracy difference).

Q5: Visual examples would make the paper stronger. For example, when sampling the different "aircraft", how different are the images from the downstream task? How do these images change when doing dataset Style Inversion?

A5: Thank you for your advice, we have added visualizations in Appendix D in our revised version, please check it.

Q6: Additionally, evaluating tasks that are specialized (like medical, satellite imaging...) that may not be as prevalent on LAION-X as "natural" images (vehicles, animals...) would make for a stronger paper and ameliorate some of the concerns expressed in the points above.

A6: Thank you for your suggestions. We conducted EuroSAT image classification under the few-shot learning regime (since the full-shot test performance is nearly 100%), and we synthesized 1000 images per class for this dataset. Here are the results.

Q7: What is the percentage of images from synthetic and real on the Mixed transfer experiment?

A7: In the mixed transfer experiments, we synthesized 1000 images per class. As for the real images per class, we refer you to Appendix A, Table 4 for a detailed description. Roughly speaking, the number of real images per class is within the range of 30~750.

Q8: In LEEP analysis in Table 1, isn't it expected that a model that has lower train/test accuracy (as seen in Figure1 for imagenet pretrained vs bridged pertaining) performs worse on this metric too? Is the difference in transferability (when tested with this metric) also present when two models compared have the same training loss?

A8: The higher the LEEP score, the better the transferability. However, we want to emphasize that the LEEP score is evaluated without any finetuning, instead, it only involves forwarding the target dataset to the pre-trained model. Therefore, our Table 1 and Figure 1 results are the same.

Thank you for your comments.

Q0: Involved data source

A0: Thank you for raising this intriguing point. We concur with the reviewer that our "Baseline and Proposed Method involve different data sources." Nonetheless, we would like to clarify that our experiments reveal that "mixed transfer" (employing both ImageNet-1k and LAION-5B) often falls short compared to "vanilla transfer" (solely ImageNet-1k). This outcome suggests that the mere expansion of data does not guarantee enhanced performance. To explicitly reiterate the primary goals of this research: 1) We delve into how off-the-shelf stable diffusion can unlock new avenues for boosting transfer learning efficacy. 2) We examine the strategic leverage of stable diffusion through our novel "bridged transfer" and "dataset style inversion" techniques.

Furthermore, we believe it is essential to highlight that our study contributes to the ongoing efforts of integrating text-to-image models into established deep learning workflows. For instance, [1] harnesses synthetic images from Stable Diffusion in place of the CC3M dataset, thereby assessing the impact on model performance with synthetic against original CC3M images; [2] employs Stable Diffusion to enrich the ImageNet 1K dataset, leading to performance enhancements against original ImageNet 1K.

Q1: Because of the above, the paper fails to answer the question of whether it is better to use large-scale image data to first train for classification (i.e. OpenCLIP) and then finetune on small data, or use large-scale image data to first train a generative image model, which can generate synthetic data to augment the small-data (as proposed). This is an important (and easy to test) question, that the paper does not resolve because of the choice of using a pretrained model on Imagenet-1k and not on the full LAION.

A1: We ran the experiments based on the reviewer’s suggestion. Note that CLIP-based tuning is different from traditional transfer learning. For example, for the CLIP-based model, researchers found that classifier finetuning outperforms full network finetuning [3], while for normal CNNs, full network finetuning works better [4]. We compare the performance of CLIP-ResNet50 with Classifier Tuning [3] and ImageNet-ResNet50 with Bridged Transfer++ on the Aircraft and Pets dataset. Here are the results with 1/8/full-shot data regime.

As demonstrated in the table, fine-tuning CLIP generally yields lower accuracy compared to Bridged Transfer++. There are several reasons for this: (1) CLIP's fine-tuning concentrates on the prompt and classifier, whereas full network fine-tuning enables the model to enhance both the feature extractor and classifier using synthetic data for downstream tasks. (2) Bridged Transfer++ gains greater advantages from an increased synthesis of images as discussed in Section 4.2, whereas CT's performance largely depends on the pre-trained model weights and a constrained set of real images.

[1] Tian, Yonglong, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. "StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners." NeurIPS 2023.

[2] Azizi, Shekoofeh, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. "Synthetic data from diffusion models improves imagenet classification." arXiv preprint arXiv:2304.08466 (2023).

[3] Wortsman, Mitchell, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes et al. "Robust fine-tuning of zero-shot models." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971. 2022.

[4] Shin, Joonghyuk, Minguk Kang, and Jaesik Park. "Fill-Up: Balancing Long-Tailed Data with Generative Models." arXiv preprint arXiv:2306.07200 (2023).

I will take into account the additional comments when reaching a consensus on the reviewer discussion period.

Still, I believe that to show that "The experimental results clearly indicate that our proposed bridged transfer framework and novel generation technique, dubbed 'dataset style inversion,' are responsible for the performance gains.", one should test using pre-trained generative models with similar scale datasets as the baseline (ideally Imagenet-1k, which are available), making an apples-to-apples comparison.

This would allow to disentangle the positive effects of 1) using large-scale data (which is well-known and unsurprising) and 2) sampling from a generative model and training with it (what is studied in the paper). Because of this, it is unclear what gains reported on the experimental section come from 1 or 2, and is misleading to attribute all the gains to 2, the method proposed. This experimental setup is the one tested in previous papers like [1] or Generative models as a data source for multiview representation learning, Jahanian et al.

Q5: The circle plots in Fig. 2 and Fig. 8 are not very intuitive to interpret. How are these produced? There is a normalization with vanilla transfer, but how are the axes shifted? For example, in Fig. 2 it seems that axes are also scaled to show mixed transfer on the inner ring, except for the Cars dataset. It would be easier to interpret the results on, e.g., bar plots or similar.

A5: Thanks for your comment. We apologize for not mentioning the axis normalization. For Figure 2, we normalize each axis so that the vanilla transfer is fixed to the 3rd ring. Under this normalization, we scale the axis so that neither the mixed transfer goes inside the 1st ring nor the bridged transfer goes outside the 4th ring. We have changed Fig. 2 to prevent confusion on mixed transfer, where we normalize the mixed transfer to the 1st ring.

For Figure 8, we normalize each axis so that the vanilla transfer is fixed to the 2nc ring. Then, we scale the axis so that the bridged transfer is fixed to the 4th ring unless the difference is less than 2%. If the difference is less than 2%, we will scale the interval to a 1% accuracy difference. We understand that It would be easier to interpret the results with bar plots, however, it may significantly increase the space of the plots. Therefore, we refrain from doing so.

Q6: It is a bit unclear what strategy has been used in figures. For example, in Fig. 2 the performance degrades with bridged transfer on DTD and Dogs, so I assume this is not with the bridged transfer++? However, in Fig. 3 and Fig. 9 bridged transfer performs equal or better on all datasets. So, here the method has been changed to the bridged transfer++? Please clarify this.

A6: Thanks for your comment. We’d like to clarify that Fig 3 and Fig 9 actually report the training accuracy rather than the test accuracy. We did not employ Bridged Transfer++ in those two figures. We intended to show that Bridge Transfer has better convergence during training, but suffers from lower test accuracy due to overfitting.

Thank you for your positive feedback on our work. Please check our detailed response below.

Q1: The main body of experiments (for example when comparing transfer learning strategies), are performed with only text prompts as conditioning for generating synthetic data. As this is expected to generate large domain gaps with the downstream datasets, it is not surprising that the mixing of real and synthetic images performs worst. How would this compare if the style inversion was included, which reduces the domain gap between synthetic and downstream data? Or using some other alignment mechanisms such as textual inversion, dreambooth [1] or diffusion inversion [2].

A1: Thank you for your question. We agree with the reviewer that the mixed transfer may benefit from DSI synthetic data. To validate this, we train the mixed transfer using either the DSI synthetic data. The results are shown below for the Aircraft and the Pets dataset.

We find that mixed transfer with DSI inversion indeed improves the performance of mixed transfer with Single Template synthesis. However, when compared to Bridged Transfer++, this method still has lower accuracy. In our view, the mixed transfer can be viewed as a type of data augmentation, which needs careful tuning to not differentiate between original images too much. Yet Bridged Transfer is a type of pre-training that has looser requirements in domain gap compared to data augmentations.

Q2: Comparisons are made to vanilla transfer (without synthetic images) and mixing synthetic and downstream data. However, it would be of interest to also add comparisons to: 1) no transfer, training from scratch on downstream data, to show the benefit of transferring, and 2) pre-training on only synthetic data, to demonstrate if there are gains in doing the 2-stage (bridged transfer), or if it's enough to only pre-train on synthetic data.

A2: Thank you for your suggestions. We agree that adding these two experiments would benefit the overall understanding of our paper. We perform experiments on Pets and Aircraft datasets, (more datasets will be added in the future). Here are the results showing (1) training from scratch, (2) vanilla transfer from ImageNet, (3) vanilla transfer from synthetic data (not ImageNet pre-trained), (4) Bridged transfer from synthetic data.

It can be found that training scratch has the lowest accuracy, and both ImageNet pretrained or synthetic data pretrained models would benefit the transfer learning. Adding those two factors together leads to our Bridged transfer results, which have the highest performance.

Q3: There should also be comparisons to vanilla transfer with FC reinit, which is not included in, e.g., Table 2 and Table 6. It seems that this makes a large difference, but it is only tested for the bridged transfer, not for the vanilla transfer.

A3: Since the ImageNet dataset has 1000 classes, which is different from the target datasets in transfer learning, we have to reinitialize the classifier in vanilla transfer. In other words, the vanilla transfer is inherently equipped with FC reinit.

Q4: It would be of interest to demonstrate examples of the dataset style inversion, to show how this is better aligned with the downstream dataset. Perhaps also some quantification of the distributional difference compared to real data, e.g. by means of FID or such.

A4: Thanks for your comment. We have added a figure in Appendix D in the revised version. Meanwhile, we have measured the FID score and demonstrated them in the table below.

Thank you for your positive feedback on our work. Please check our detailed response below.

Q1: More analysis on benefits of synthetic images

A1: We quantitatively convert the effect of synthetic data to the effect of real data by studying this question in few-shot learning scenarios. We run the vanilla transfer from 1-shot to 40-shot one by one and match the accuracy of the bridged transfer. Therefore, we can evaluate how much data the vanilla transfer would need to realize the same accuracy as the bridged transfer. The difference between vanilla and bridged transfer is the gains of synthetic data. The results for the Aircraft dataset of ResNet-18 are shown below:

Significantly, our findings reveal that augmenting a dataset with 3,000 synthetic images per class can be as effective as acquiring 24 new real images per class, an impactful insight considering our initial dataset comprises just 16 images per class. From a cost perspective, this is noteworthy: commercial image procurement may range from 1 to 3 dollars per image, whereas leveraging stable diffusion to generate 3,000 images entails a mere $1.50, as documented by AWS's cost analysis (https://aws.amazon.com/blogs/machine-learning/maximize-stable-diffusion-performance-and-lower-inference-costs-with-aws-inferentia2/). This cost efficiency could represent significant savings for researchers and commercial entities alike.

From the standpoint of privacy protection, synthetic image generation offers a substantial advantage. Traditional methods involving web-scraped or manually collected images mandate rigorous examination for personal identifiers and sensitive content, a process that can be both time-consuming and prone to error. In contrast, synthetic data not only circumvents many of these privacy concerns but also enables a mathematical assessment of potential risks. Studies such as [1, 2] have explored the concept of Differentially Private Stable Diffusion (DP-SD), suggesting that models trained on DP-SD-generated data inherit the framework's privacy-preserving attributes, thereby enhancing the overall security of the information.

[1] Lin, Z., Gopi, S., Kulkarni, J., Nori, H. and Yekhanin, S., 2023. Differentially Private Synthetic Data via Foundation Model APIs 1: Images. arXiv preprint arXiv:2305.15560.

[2] Sehwag, V., Panda, A., Pokle, A., Tang, X., Mahloujifar, S., Chiang, M., Kolter, J.Z. and Mittal, P., 2023. Differentially Private Generation of High Fidelity Samples From Diffusion Models.

Q2: Beyond the transfer learning paradigms, will this observation also hold in other learning paradigms?

A2: Yes, we can apply synthetic images to a randomly initialized model (no real data pre-trained), and then transfer to target datasets. Here, we test the performance on Aircraft and Pets datasets. The results are shown below:

It can be found that our method can effectively improve the performance of a randomly initialized model. On the Aircraft dataset, the synthetic data even outperforms the ImageNet pretrained models.

Q3: Beyond text-to-image generative models, will the idea of bridged transfer also work?

A3: Thanks for the question. Generally, we think bridged transfer will also work on other types of synthetic data, like GAN-based synthetic data. However, given the time limit in the rebuttal period and the synthesis model is a little out of our scope, we'd like to leave it as our future work.

Q4: Also, it might be helpful for the reader, if there was a subsection/paragraph that describes the difference between Bridged transfer and Transfer1. For example, Bridged transfer uses Dataset style inversion to reduce the domain gap whereas Transfer1 leans on filtering approaches to create a higher-quality dataset.

A4: Thanks for your suggestion, we have added a section in Appendix E showing the difference between our work and Transfer 1. In the previous experiments, we conducted experiments under the Transfer 1 framework and we believe these two papers share similar observations in leveraging the synthetic data. In addition to Transfer 1, our work studied how to more effectively utilize synthetic in the general transfer learning framework. We find that synthetic data promotes feature representation learning but may bias the FC layer. We will add these inspiring discussions to the main paper when we have more dataset results.

Q5: It might be helpful to see some qualitative results on the benefits of data style inversion (DSI). It can visually show that the domain gap of synthetic datasets with target dataset decreases (for e.g. comparison between with and without DSI module for the same text prompt in the context of some target datasets)

A5: Thanks for your comment, we have added a figure in Appendix D to show the effect of DSI. Please check it.

Q6: It might improve the readability of the paper if Bridged Transfer+FC Reinit & mixup is also included in figure 2.

A6: Thank you for your suggestion. We have added the Birdged Transfer++ (Bridged Transfer+FC Reinit & mixup) results into Fig 2.

Q7 How much training time/wall-clock time and memory resources does it take for dataset style inversion (DSI) for each dataset to optimize the S* token? For e.g. it’s mentioned that it takes 20k training iterations for the Aircraft dataset.

A7: For all datasets, we optimize the token for 20k training iterations (3 hours on one A100 GPU). Hence, the training wall-clock time is the same across different datasets. Note that when compared to class-wise textual inversion, the speed-up ratio will be correlated with a number of classes in that dataset.

Q8: Are there any qualitative reasons that for datasets like DTD and Pets, the improvement is kind of limited ? Does the synthetic dataset has higher domain gap or is this a fine grained task for which text-to-image models can’t generate enough informative samples. Also, it would be interesting to see the impact of synthetic data volume increase for these classes similar to Fig 5(a).

A8: During our experiments, we found that there could be two reasons that make DTD and Pets hard to improve with synthetic data. (1) The domain gap between synthetic and real data is too huge, or the label cannot describe the class very well. For example, in the DTD dataset, the label name is the adjective of each texture, e.g., “banded, braided”, which can correspond to lots of different banded textures and braided textures. In this case, we have to use DSI to mitigate the gap. (2) The second reason is the original datasets contain enough data samples for recognition, adding much more synthetic data might lead to severe overfitting, e.g. Pets. In this case, we can observe much more improvements in few-shot transfer, while moderate improvements in full-shot transfer.

Q9:Is Bridged Transfer++ equal to “Bridged Transfer + FC Reinit & mixup” or “Bridged Transfer + FC Reinit & mixup + DSI” ? In table 3, does Single Template correspond to “Bridged Transfer + FC Reinit & mixup” ? If yes, why are the number for DTD not matching up between Table 2 (72.3 +- 0.3) and Table 3 (71.5+-0.4) while the numbers does match for rest of the dataset like Aircraft, Cars, Foods and SUN397 ?

Q10: In table 3, what does Single Template correspond to (which Bridged Transfer version and what’s the network architecture)?

A9/10: Thank you for your question. We apologize for the unclear description and wrong numbers in Table 3. In Table 3, the Single Template uses ResNet-18 as the architecture and both Single Template and DSI use Bridged Transfer ++, which is Bridged Transfer with FC Reinit and mixup. We revisited our experiments and found that we forgot to add a mixup loss function when running DTD datasets. We have changed the accuracy to new ones.

While it is challenging to draw an apple-to-apple comparison between Transfer 1 and our method, we have endeavored to adapt both for comparison: 1) We employed Stable Diffusion with RF and RG for Transfer 1, based on its official code, and 2) We performed bridged transfer (limited to the classifier) with CLIP. It's important to clarify that this experiment is not about directly contrasting the performance of Transfer 1 and our method in their unaltered states, but rather to understand how they operate within different configurations. Due to the time limit in the rebuttal period, we select Aircraft and Pets datasets to test the performance.

Based on the results, we have several common observations with Transfer 1 and several new findings on top of it.

1. In few-shot recognition, Real Guidance synthesis is always better than Basic synthesis, which is also observed in Transfer 1.

2. However, in 8-shot setting, we found that Basic synthesis + Mixed (Transfer 1) cannot improve the vanilla transfer, this is similar to our paper’s finding that regular synthetic data has a distribution shift and may lead to accuracy decreases in mixed transfer.

3. The bridged transfer does not always perform better than the mixed transfer using the Transfer 1 framework (CLIP, classifier fine-tuning) which is also discovered by Transfer 1 in Table 7. We think the reasons are twofold: (1) Transfer 1 uses classifier tuning of the CLIP model, while our pipeline uses whole network tuning. In our paper, we found that synthetic data may lead to bias in the final classifier layer, therefore, we employ the FC reinitialization in bridged transfer++. In Transfer 1, since only the classifier is allowed to be tuned, it is not surprising that bridged transfer may have lower performance than mixed transfer when only optimizing the classifier. (2) Transfer 1 uses different batch sizes to balance the synthetic data and real data (512 for synthetic data, 32 for real data) so that the network weighs more in real data to prevent distribution shifts. This is why Transfer 1 needs to freeze BN, while we do not need to freeze BN layers in bridged transfer. Nonetheless, when we apply our bridged transfer approach to ResNet-18, it yields enhanced accuracy on the challenging fine-grained classification dataset 'Aircraft', despite ResNet-18 being significantly smaller than CLIP. Moreover, Transfer 1 exhibits superior performance on the 'Pets' dataset. These outcomes corroborate our earlier assertion: although Transfer 1 and our method focus on different models and scenarios, they both effectively demonstrate the potential of synthetic data to boost accuracy for their respective settings and tasks.

We hope our response can sufficiently address your concerns.

Thank you for your comments and suggestions to improve our paper. We would like to reply to your questions and suggestions in detail.

Q1: Performance of Transfer1 on the same setting (i.e. having same generative text2image model, same evaluation datasets and using the proposed approaches of dataset filtering RF, RG to close the domain gap)

Q2: Performance of Bridged transfer on pre-trained CLIP models and then comparison with Transfer1 on the same proposed evaluation setting introduced by Transfer1.

Q3: The idea of the above 2 suggestions is to glean out what additional information we can learn on top of the findings discovered by Transfer1 approach.

A1/2/3: We are grateful for the recommendation to juxtapose our method with Transfer 1. While acknowledging the importance of Transfer 1 in utilizing synthetic data for image recognition, we aim to delineate the distinctions and similarities between our approaches. We must emphasize that Transfer 1 utilizes classifier tuning tailored for the CLIP model; conversely, our work concentrates on whole network tuning applicable to a spectrum of models, such as ResNets. It is crucial to recognize the significant differences in pre-training, fine-tuning, and application methods between the models assessed by Transfer 1 and those in our study. For instance, it has been observed that fine-tuning an entire CLIP model may detrimentally impact its robustness and accuracy [1]. Consequently, not all insights from Transfer 1 are directly transferrable to our proposed bridge transfer frameworks, and vice versa—a point we will substantiate through our experiments. A direct comparison between the tuning of CLIP and general models falls beyond the scope of Transfer 1 and our work.

In this study, we delve into transfer learning for general models, as this scenario is not only prevalently employed in practical settings but also merits investigation to comprehend how synthetic data can enhance these applications.

[1] Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., ... & Schmidt, L. (2022). Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7959-7971).