The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better

Thank you for your valuable feedback! We are honored to hear that you enjoyed our paper, and are grateful you found our research question “timely” and “totally overlooked” by our community.

Your question—whether synthetic images can offer unique advantages for tasks with rare or complex concepts—is very exciting. Such concepts are precisely the concepts for which collecting labeled real data is difficult, and thus exactly where we have the greatest need for synthetic data. These directions are exciting topics for future work that we will discuss further in our final paper. For example, synthesizing very complex concepts might be made feasible by leveraging the compositional generalization abilities of generative models, which recent research promisingly suggests may be a unique boon of generative models compared to other models trained on the upstream data [1, 2] (references below).

Just for curiosity’s sake, we also performed a preliminary study to try and relate the concept-wise performance of models trained on synthetic or retrieved data with the rarity of each concept. Specifically, given a concept denoted via a text string $c$ (e.g., “Airbus A320”), we approximate that concept’s frequency in LAION-2B by counting the number of image-text pairs with text containing $c$ as a (case-insensitive) substring. We scattered the concepts with their frequency on the $x$-axis and model accuracy on the $y$-axis. Unfortunately, we were not able to find any clean trends — with existing methods, the performance gap between synthetic and retrieved data does not appear to systematically decrease on rarer concepts. To speculate, this may be because generative models are trained on the same data pool that we retrieve from, and thus may also have difficulty learning rare concepts during general pretraining as they are seen less frequently. Recent work [3] corroborates this conjecture; however, we note there are asterisks to our preliminary study, as we did not extensively validate the robustness of our concept frequency metric. Nonetheless, if off-the-shelf pretrained generators are indeed less effective at generating rare concepts, a potential future direction would be to resolve this limitation by taking advantage of a small set of examples to adapt the generative model in a sample efficient manner (e.g., perhaps through textual inversion). Afterwards, we may be able to compose the concept with the generator’s existing knowledge to sample new data variations.

Thank you again for your time, and thank you for drawing our attention to these exciting future directions that may lead to gains beyond our retrieval baseline. We will discuss them thoroughly in the next version of the paper!

• [1] Your diffusion model is secretly a zero-shot classifier. Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak.

• [2] Text-to-image diffusion models are zero shot classifiers. Kevin Clark and Priyank Jaini.

• [3] No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance. Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H.S. Torr, Adel Bibi, Samuel Albanie, Matthias Bethge.

Thank you very much for your time and effort in this process! We appreciate your insightful comments, and will be sure to incorporate all other reviewer's valuable feedback in the next version of our paper. Thank you for helping make our work stronger!

Thank you for your valuable feedback! We are excited that you found our proposed baseline for measuring the utility of synthetic data to be “novel and inspiring.”

Your feedback prompted us to perform additional experiments to further validate our paper’s findings, and we believe the new results corroborate our paper’s main finding that state-of-the-art synthetic data training methods lag our proposed retrieval baseline. Thank you for helping us strengthen our paper! We will include all new results in the next version of our paper, and describe the details below to address each of your points.

Q1 (CLIP score distributions post-filtering): Do the distributions of CLIP similarity scores in the final synthetic and retrieved training datasets significantly differ? Are retrieved and synthetic images ranked and filtered separately, or together?

A1: Great question! Significant differences in CLIP similarity could potentially explain gaps in downstream training performance and is worth studying. In our paper, we ranked and filtered synthetic and retrieved training data separately (i.e., to keep the experimental design as clean as possible, we do not use any information from the retrieved images to inform the selection of synthetic data, and vice versa). We use per-class score thresholds for filtering, which we found empirically beneficial for both synthetic and retrieved data. We histogram the CLIP similarity score distributions of the resulting filtered training data in Figure R2 of the rebuttal PDF. Overall, despite setting the filter score threshold for synthetic and retrieved data independently, we find that the distribution of post-filtering synthetic image CLIP scores is right-shifted compared to the distribution of post-filtering retrieved image CLIP scores. In other words, synthetic images have comparable or even higher CLIP scores than retrieved images on average; CLIP judges synthetic data to be higher quality on average. Thus, CLIP score differences alone do not explain the lagging training performance of synthetic data.

Q2 (Filtering out images with artifacts): As shown in Figure 3, synthetic ‘flute’ images have obvious generation artifacts. These images should be filtered for fair comparison.

A2: To clarify, the artifact-afflicted images shown in Figure 3 are post-filtering. Even though the ‘flute’ images of Figure 3 are obviously wrong to humans, CLIP assigns them relatively high scores of $0.285, 0.265$, and $0.263$. For reference, a CLIP score of $0.249$ reflects the top $30$% of all retrieved ‘flute’ images. Furthermore, we found obviously wrong and artifact-ridden synthetic ‘flute’ images that have even higher CLIP scores $> 0.3$, placing them in the top $5$% of all retrieved ‘flute’ data.

Our paper adopts CLIP filtering since it is the current best synthetic image filter [22] we are aware of; in fact, many recent works forgo data filtering altogether [3,21,51,57,58] despite its positive impact on training performance [22] (our study corroborates this gain; CLIP filtering synthetic data improved ImageNet zero-shot accuracy by 1.03% and LP by 0.91% over no filtering at 4M scale). We aim to ground our argument in the current state of synthetic training data research, so we do not explicitly innovate new data filtering methods beyond CLIP filtering. Nonetheless, these findings spurred by your feedback makes it apparent that CLIP filtering is limited. We will include a more detailed discussion of this point in our final paper to motivate future filtering methods.

Q3 (Alternative prompts for synthetic data generation): Synthetic images may suffer in diversity when the generation prompts are fixed. Other prompting strategies should be considered, such as using LAION alt-text or outputs from a captioning model.

A3 Thank you for the point! First, to clarify a potential miscommunication, the prompts used to generate synthetic images in our work are not fixed – rather, each image prompt is sampled from a probabilistic large language model (LLM) that is tasked to include additional relevant knowledge about the desired category (e.g., for the “dog” category, a sampled generation prompt might be “a photo of a dog playing with a ball in the park”). We will make this more clear in our paper.

We adopt this LLM-guided prompting strategy from SynCLR [57], which shows that it outperforms many alternative prompting strategies at the hundred-million scale, including prompting with LAION alt-text. We did not find any existing comparison between LLM-generated prompts and prompts from an image captioning model as proposed by the work you referenced [1], so we conducted this study ourselves.

Starting from the unfiltered retrieved training set, we used BLIP-2 to caption each image and construct prompts of the form “a photo of {classname}, {BLIP-2 image caption}” following [1]. We then performed top-30% CLIP filtering on the resulting synthetic images. We compare the performance of training with filtered synthetic images generated with three prompting distinct strategies to our filtered retrieved data in Figure R3 of the rebuttal PDF. Specifically, we compare training with filtered synthetic images generated from (1) our original LLM prompts (orange lines), (2) BLIP-2 captions (red lines), and (3) LAION alt-text from retrieved data (green lines). Overall, among the three generation strategies, our original LLM prompts perform best on ImageNet, and perform comparably to BLIP-2 captions and LAION alt-text on Aircraft. All three synthetic strategies lag the performance of retrieved data. Thus, our conclusion—that existing synthetic image training methods do not surpass the retrieval baseline—holds under these variations of image generation strategy.

We were not aware of [1] at the time of submission — thank you for pointing it out to us! We will include a reference to that work and the above experiments motivated by [1]’s captioning method in our final paper.

Thank you for your time and continued interest in our work! We apologize for our confusing formatting — our initial response to your last question is folded into Q3/A3 of the rebuttal text above. For your convenience, we summarize the relevant parts here:

• First, to clarify a potential miscommunication, our generation prompts are not fixed, but rather sampled from a probabilistic LLM that is tasked to generate image captions for each desired visual concept. We use these LLM captions as text-to-image prompts.
• We adopt the LLM-guided prompt method from SynCLR [57], a current SOTA work which shows that synthesizing images with LLM prompts can outperform alternative generation strategies (including synthesizing images from LAION alt-text) at hundred-million scale.
• We experimentally compared the training performance of filtered synthetic images generated using either (a) our original LLM prompts or (b) the LAION alt-text of our retrieved images. We report results of this new scaling experiment in Figure R3 of the rebuttal PDF. Synthetic images generated via LLM prompts outperform images generated with LAION alt-text prompts on ImageNet, and perform comparably on FGVC-Aircraft. Regardless of generation strategy, synthetic images lag the retrieval baseline.

We missed the part of your question about the diversity of retrieved versus synthetic images in our initial response. We sincerely apologize for the confusion! To clarify your question, we further compared the image variation in our final filtered Aircraft and ImageNet adaptation datasets, which consist of either (a) retrieved real images, (b) synthetic images generated with our original LLM prompts, or (c) synthetic images generated from the LAION alt-text of the retrieved images. We quantify image variation via the average pairwise cosine similarity of the CLIP image features for each dataset (i.e. lower average similarity → higher variation). To understand variation at large scale, we perform this analysis on the largest-sized version of each dataset . Results are as follows, with specific scale size in parentheses:

Overall, your intuition is correct: synthetic images generated with LLM prompts indeed exhibit higher average pairwise cosine similarity compared to retrieved images, suggesting deceased variation. This gap persists even though the scales of the LLM prompt synthetic datasets are significantly larger than the scales of the retrieved datasets (e.g. 4M vs 2.5M for ImageNet, 500K vs 139K for Aircraft). Moreover, generating synthetic images based on the LAION captions of the retrieved images does improve the measured variation of the resulting synthetic images.

However, interestingly, improvements in synthetic image diversity alone do not directly translate into significant improvements in downstream model performance. As shown in Figure R1, models trained on the less-diverse LLM prompt generated images we originally considered perform better or comparably to models trained on the more-diverse LAION alt-text generated images. Nonetheless, we are excited that the experiments you suggested have uncovered another axis along which synthetic and retrieved images differ. We will include these experiments and detailed discussion in the updated version of our paper to help motivate future synthetic data work.

Thank you for your valuable review! We highly appreciate your feedback. To ensure that we can thoroughly address the review, could you please clarify what works [1] and [2] refer to? It seems that [1], [2] in the review text do not match references [1], [2] in our paper. Also, to double-check, [57] in our paper is a citation to SynCLR -- does "StableRep [57] implicitly..." in the review text refer to SynCLR, or does it refer to a different citation? Thank you very much for your time!

[1] Generative models as a data source for multiview representation learning. Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola" (GenRep)

[2] Using diffusion models to generate synthetic labeled data for medical image segmentation. Daniel G Saragih, Atsuhiro Hibi , Pascal N Tyrrell

[3] StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners. Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, Dilip Krishnan

[57] corresponds to the submission citation (SynCLR). All three GenRep [1], StableRep [3] and SynCLR [57] "implicitly have the baseline proposed in the paper", by using the entire dataset (i.e. all the neighbors). In the three papers, the networks are pretrained with generative samples, and the full datasets are used to train the generative models. [1] does not outperform the baseline, while [3] and [57] do.

Thank you for your valuable review! We are glad you found our research question “highly relevant” for both vision and NLP.

Before answering your points directly, we’d like to clarify a possible miscommunication. A key point in the review is that our proposed retrieval baseline already implicitly exists in other work. For example, SynCLR [57] compares synthetic images against the generative model’s full training set (i.e. retrieve everything) and finds that synthetic data performs better.

While such results are exciting, this existing baseline—comparing synthetic data to the generator’s full train set—is distinct from our proposed baseline, and is insufficient for measuring the true added value from the generative model itself. When we sample synthetic training data, we often sample data that is targeted to tasks we want our learner to perform well. Works like [57] use this idea to generate synthetic data that beats the full real dataset for specific tasks. However, by comparing to the full real data, a task-agnostic distribution of images, [57] and other works [21] implicitly conflate the effects of training on synthetic versus real data with the effects of targeted data sampling. We aim to disentangle these factors and answer: are observed gains from synthetic data truly due to added information from the generator? Or is it since the synthetic data is implicitly sampled in a targeted manner from large upstream real datasets? To study this, we construct a retrieval baseline that explicitly controls for targeting (L39-45), thus overcoming a key limitation of prior work.

Overall, we are excited by the same possibility you highlighted—that synthetic data can transfer useful information from the generator's implicit biases. However, to ensure our field progresses toward this goal, we must carefully measure the current state of synthetic data methods. We believe our work is a crucial step in this direction.

Q1: “The main weakness of the paper is that… Although the question studied is really relevant to the community… the technical contributions of the paper seem limited.”

A1: Our goal is not to innovate new methods; rather, we seek to introduce a conceptual baseline that helps our field better understand the true utility of synthetic training images. Our proposed baseline can be implemented with existing image retrieval techniques; we aim to keep our baseline simple to facilitate comparison against it.

Q2: Unclear wording of data processing inequality sentence.

A2: Thanks! We will revise to discuss that SynCLR [57] is indeed motivated. We agree that synthetic images can contain useful information from the generator’s training process. However, it is unclear if [57]'s gains are truly due to this added information—[57]'s gains may also be from targeting the synthetic data to the evaluation tasks. Our retrieval baseline controls for the effect of targeting, allowing us to better measure whether the observed gains from synthetic data are due to the generator’s added information.

Q3: Why only analyze Stable Diffusion (SD), why not analyze closed-source models to see if they suffer similar limitations?

A3: We analyze SD so we can contextualize its analysis with its performance relative to our retrieval baseline, which is only possible with open data. In contrast, if we find some closed-source generative model G yields higher quality images than SD, it would be unclear whether these gains in image quality reflect improvements in the information added by G to the synthetic images, or whether G was simply trained on a higher quality dataset than SD. We also aim to keep consistency with recent work [21,51,57] that all use SD as the generator. We will revise to clarify these points.

Q4: The proposed baseline exists in prior works, which compare synthetic data against the generator’s full training set.

A4: Please see our discussion above for a detailed response. Briefly, SynCLR [57] does not control for the effects of data targeting, and GenRep [1] samples data from unconditional GANs, whereas modern synthetic data is sampled in a targeted manner. Our retrieval baseline which controls for targeting is now necessary to understand synthetic data gains in the modern regime. StableRep [3] only compares against a small random subset of the full train data. We will clarify in our final paper. Thank you!

Q5: The comparison is against a simple way to generate synthetic training images, but works like SynCLR [57] propose more advanced ways that can outperform CLIP.

A5: To clarify, our paper exactly adopts [57]’s synthetic data method. [57] only significantly beats CLIP on the Aircraft task, which our results corroborate. Sampling data from generators trained on data-scarce domains as you cited [2] is an exciting idea that we will discuss in related work; we focus our study on sampling data by prompting web-pretrained models like Stable Diffusion to be consistent with other work [21,51,57].

Q6: The high performance of retrieved data may be due to training set contamination.

A6: Good point! The generator is trained on the same data we retrieve from, so in theory synthetic data can also have benchmark train set contamination. We agree this contamination is more direct when we retrieve data; we thus further decontaminated all retrieved data for the train sets following [18]. We report the amount of removed data in Table R1 and plot the results of training on decontaminated retrieved data in Figure R1 of the rebuttal PDF. Overall, while some retrieved images were indeed similar to train set data (an average 1.9% of retrieved data was removed), discarding them minimally impacted performance.

Q7: Confusing zero-shot terminology.

A7: Thanks! We will clarify in our paper: our zero-shot models are not trained on downstream benchmark data. We finetune CLIP on retrieved or synthetic data only and evaluate the resulting model as-is on the test set.

Thanks for the rebuttal. Could authors give a bit more details about what they mean by targeted refinement and why [57] and [21] "implicitly conflate the effects of training on synthetic versus real data with the effects of targeted data sampling". I still struggle to see the difference between [57] and "targeted refinement", is it simply that [57] targets the full training dataset instead of a subset like the case studied? Why is this substantially different?

Thanks for the detailed clarification, you are right! If I understand correctly, and as specified in Section 4.1, SynCLR does create a LAION-like set of captions, and they train on all the captions once (using CLIP), and not retrain per each downstream task. Although this is less severe than training specifically for each downstream application (i.e. only training on Airplanes when the downstream is also Airplanes), it is true true that during the process of creating captions they use all the captions from all downstream tasks (with many variations and augmentations), so indeed, their process conflates the effects of the synthetic sampling and the targeting of (all) the downstream tasks they study.

After the discussions, authors have addressed my concerns, and I'm raising my score to acceptance. Still, I expect the final version of the manuscript to include clarifications on the mentioned points.

Thank you very much for your time and effort in providing us feedback! We highly appreciate your review. Could you please clarify your point (1) under the weakness section? We would love to address your points as throughly as possible. Specifically, we were hoping to clarify the following:

"... downstream fine-tuning results of the real-and-synthetic data of non-training data. And not clear about the advantage of using upstream real data as baseline but not the othe dataset."

By "non-training data," do you mean that data outside the generative model's training dataset should also be used as the baseline? Does the main question under the Question section ("There is a main confuse...") refer to the same point? Thank you so much for your time!

Thank you for your valuable feedback! We are glad you found our experimental setup “robust” and our discussion to “have a direct impact on the development” of future models. We address each of your points below.

Q1 (Why retrieval baseline?): Why emphasize the importance of retrieving data from the generative model’s training data as an evaluation baseline for synthetic data? Why adopt the retrieval baseline over simply comparing synthetic data to high-quality data outside the generative model’s training set?

A1: Great question! Our research goal is not just to study whether synthetic training data can improve model performance (which as you noted, would not require the retrieval baseline). Rather, we seek to understand where performance gains from synthetic data come from, and to contribute a principled way of measuring whether model-generated synthetic data can surpass the real data it derives from (i.e., the data used to train the generator). To study this question empirically, we propose the retrieval baseline. This baseline enables us to disentangle whether gains from synthetic data are due to (a) the fact that synthetic images are implicitly subsampled from the generator’s huge real training set, or (b) due to the generator truly adding useful information beyond its training data. If we compare synthetic data to high-quality data outside the generator’s training set, then these two factors cannot be disentangled. For example, even if a synthetic data method outperforms high-quality outside data, the gains from synthetic data may simply be due to the generative model seeing higher-quality and larger-scale data during pretraining. In contrast, our baseline of retrieving real data from the generator’s training set controls for the effect of information from the upstream real data. Retrieval also controls for the effects of targeted data sampling (i.e., synthetic data is often targeted to specific tasks), which has been conflated with synthetic data in prior work [57].

We believe the research questions we pose—how to measure if/when synthetic data surpasses the real data it was trained on—are timely to tackle. There is surging interest in building SOTA vision models with synthetic data [3,21,51,57,58], and many works now show strong gains. We find that such gains—while exciting—still fall short of our retrieval baseline, suggesting that today’s synthetic data methods do not provide significant information beyond the generator’s training data. Hence, we hope our baseline will set a strong and simple target for the field going forward. We further discuss how our paper can impact future work in response to your Q4 below. We will clarify all points in our final paper. Thank you for helping us strengthen our work!

Q2 (Insufficient filtering): The CLIP score used in the paper to filter synthetic images is not sufficiently sophisticated and does not reflect current best filtering practices.

A2: Thanks for the question! This point may be mistaken. While we agree that improving data filtering is an exciting future direction for improving the utility of synthetic training data, filtering based on CLIP score remains state-of-the-art [22]. In fact, many recent representative works do not apply any quality filters whatsoever to generated synthetic training images [3,21,51,57,58]. Our work uses CLIP filtering to optimize the performance of synthetic data, which [22] finds improves training performance over no filtering. Our study corroborates [22]’s finding; for example, applying CLIP filtering to synthetic training data improved ImageNet zero-shot and LP accuracy by 1.03 and 0.91 points (respectively) over no filtering at 4M data scale.

Q3 (Retrieval is impractical): Retrieval from a generator’s training data is often not practical due to privacy concerns or closed-source data, which may limit the work’s applicability.

A3: Good point! We make a similar note and discuss its implications for future research in the introduction and discussion of our paper (L70-73, L312-315). While privacy concerns and closed data are practical reasons to use synthetic data over retrieved data, these application constraints are orthogonal to our research question. Specifically, our goal in proposing the retrieval baseline is not to prescribe a general method for maximizing downstream task accuracy (which, as you noted, would require consideration of downstream constraints), but rather to advance our understanding of what value synthetic data provides beyond the training set of its generator.

Q4: How can this research and the proposed retrieval baseline inspire future work?

A4: We believe our work can inspire future research in two primary ways. First, comparison against our retrieval baseline provides a principled target for future synthetic data research to aim for. If any synthetic data method outperforms training on data retrieved from the generative model’s training data, then that is strong evidence that the generative model has added (e.g., through its inductive bias, through its compositional abilities) additional useful information on top of the data that it was originally trained on. In contrast, as discussed in A1, if a synthetic data method outperforms some other source of real data outside of the generative model’s training set, then we cannot draw a similar conclusion.

Second, by demonstrating that retrieving real data from the generator’s training set currently surpasses synthetic data, we hope to sharpen the field’s intuition of when existing synthetic data methods can be useful in the present day. Many recent works position synthetic data as a potential drop-in replacement for real data, even when the retrieval baseline is possible. Our findings do not corroborate such positioning; instead, existing synthetic data methods are most promising when downstream constraints (e.g., privacy concerns) prevent the retrieval baseline from being realized.