Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World

Thank you for your thoughtful review and recommendation. We appreciate your recognition that our paper is well-written and that our claims are properly validated.

Theoretical Results for Accumulate-Subsample

You raise an excellent question. The accumulate-subsample workflow presents significant analytical challenges because of subsampling the data at each model-fitting iteration. We welcome any suggestions you might have for tackling this theoretical challenge, as we believe it represents an important open problem for the field.

Clarification of Relationship to Dohmatob et al. in Section 4

Could the authors be more clear on how the theoretical results in Dohmatob et al. differ from the insights shown (experimentally) in Section 4?

Dohmatob et al. 2024 show in Corollary 3.3 that in a simplified mathematical setting, the test error on real data scales as $(T_{real} + T_{AI})^{-(1 - 1/\beta)} + k^{-(\beta -1)}$. This means that in this setting, AI-generated data can help improve performance. Directly comparing our empirical results to their theoretical results is difficult because our generative models differ. Additionally, their result decouples the amount of data $T_{real} , T_{AI}$ from the tail cutoff $k$, whereas decoupling these two factors in real data would be difficult (and potentially not possible). We see our results as complementary, but different, and want to give appropriate credit to prior work. We will clarify these distinctions in the revised paper.

Realistic Settings

Can the authors provide results for a more realistic setting than the one used in the gaussian and kde experiments. The SFT one conducted in the paper is a good example of realistic scenario.

We appreciate this feedback. To explain why we used these classical statistical models, at least three previous papers have studied multivariate Gaussians, making this setting crucial for consistent comparison against their results. These settings offer analytical tractability for theoretical results while still demonstrating the key phenomena.

We’d also like to highlight that our paper contains results from pretraining sequences of language models on real text (TinyStories), as shown in Figure 4.

Could you please clarify what "realistic" properties you're looking for that our current experiments don't address? Is it a matter of the particular generative model? If so, what model and data combination would you like us to run experiments for?

We'd be happy to expand the language model experiments with additional details in the revision or to conduct new experiments if you have specific suggestions.

Definitions of Model Collapse

You raise an excellent point. A recent review (https://arxiv.org/abs/2503.03150) identifies multiple definitions of model collapse. We focused on test loss divergence as it addresses an existential threat: all future generative models becoming useless. We'll expand our discussion to acknowledge alternative definitions and clarify our specific focus.

Novelty in Accumulate-Subsample Paradigm

We appreciate the opportunity to clarify what distinguishes our work from prior research:

1. Practical Framing: We specifically examine accumulate-subsample through fixed compute per model-fitting iteration, which directly models real-world ML development constraints. This framing is absent from the papers cited but is absolutely critical. One can study all manner of different workflows for data, but their importance depends on how closely they model the real world.

2. Training Methodology: Bertrand et al. (2024) studies iterative retraining, where each model is initialized from the previous model and each optimizer is initialized from the previous model’s optimizer state. The iterative retraining approach in Bertrand et al. mitigates model collapse because models are constrained from wandering too far from their predecessors. However, this doesn't reflect how frontier models are developed in practice. In comparison, we model the scenario where successive model generations are trained independently.

3. Data Evolution: We consider synthetic data accumulation across multiple model generations, whereas prior work typically only considers synthetic data from the most recently trained model e.g., Bertrand & Alemohammad.

We will improve our manuscript to clarify.

Related Work

How does this work differ from Dohmatob et al. (2024), which already studied real-vs-synthetic data trade-offs?

Our work differs substantially from both Dohmatob papers. One examines linear regression specifically, while the other investigates scaling laws. Both are largely theoretical. In contrast, our work is more empirical and spans multiple generative paradigms (Gaussian modeling, KDE, language model fine-tuning) and examines different data treatment workflows across these diverse settings.

Please move the related work section to the main text.

If accepted, we will move Related Work into the main text after the Introduction. We placed it in the Appendix due to the 8-page limit.

We will expand our related work to include additional relevant papers discovered during the review process e.g., Gillman et al. 2024 and better clarify the positioning of our contributions.

Understanding Accumulate-Subsample

Can the authors justify why accumulate-subsample prevents collapse beyond empirical results?

The accumulate-subsample setting is difficult to study analytically and we were not able to prove satisfactory results. Intuitively, accumulate-subsample slows collapse because each new training batch contains a fraction of real data that provides an "anchor" to the true distribution.

Benefits of Synthetic Data

Our contribution isn't showing that synthetic data can help performance - we agree this possibility is well-known across many papers. Rather, one contribution is that such a result holds for kernel density estimation - a simple yet fundamental statistical model - without sophisticated data manipulations. Another contribution is assessing the role of cardinality or proportionality of synthetic data , which to our knowledge was previously unexplored.

Upon careful review of Seddik et al. (2024), we couldn't find where they demonstrate synthetic data improves performance.

Realistic Data Quality

Our experimental design deliberately aimed to model how synthetic data proliferates on the internet in practice:

• Content on the internet is not systematically filtered. Vast amounts of web data is garbage, spam, AI-generated slop, etc.
• By studying the "pessimistic" scenario without filtering, we establish a baseline understanding of collapse dynamics. Filtering can only improve the quality of models.
• That said, we agree data quality is an important factor, and we view our work as complementary to research on data quality research. Indeed, we write "Our experiments take a pessimistic viewpoint, in the sense that our experiments pay no attention to the quality of data, whereas in practice, engineers heavily filter data based on various indicators of data quality, e.g., (Brown et al., 2020; Lee et al., 2023; Wettig et al., 2024; Penedo et al., 2024; Li et al., 2024b; Sachdeva et al., 2024); for a recent review, see Albalak et al. (2024)."

There is no long-term stability analysis, making it unclear how performance evolves after many more iterations.

We ran most experiments for 100 iterations across most settings. Running language modeling pretraining for longer proved beyond our limited compute budget. We're open to extending our experiments.

On Novelty

My biggest concern with the paper is its novelty. It builds heavily on Gerstgrasser et al. and section 2 (a large part of the paper) only provides new results for different kinds of generative models with equal conclusions.

We disagree with this characterization. Our Section 2 identifies and resolves a fundamental tension in the literature where competing papers have reached contradictory conclusions about model collapse. This systematic evaluation across multiple generative settings provides valuable scientific evidence about the generality of these phenomena.

Moreover, we make several fundamental contributions:

1. We discovered that in KDE settings (Figure 2), certain combinations of real and synthetic data actually outperform models trained on real data alone - a counterintuitive result not previously reported for KDEs. KDEs are appealing because they are a widely used statistical model that are straightforward to experiment with and analytically tractable to study.
2. We provide novel theoretical results (Theorem 1) on the limiting distribution for univariate Gaussians under the accumulate training workflow, proving that both the covariance and mean errors converge to non-zero constants - in stark contrast to the replace setting where they diverge. This theoretical result mathematically characterizes why the accumulate workflow avoids collapse.
3. Our work is the first to systematically compare all three workflows (replace, accumulate, accumulate-subsample) across five different generative settings, providing a more comprehensive understanding of when and how model collapse occurs.

On Accumulate-Subsample and Prior Work

The second part of the paper (accumulate-subsample) provides empirical insights that mixing real and generated data slows down model collapse. This has been observed and discussed in literature as early as Bertrand et al., Alemohammad et al. and even for the case of language models in Briesch et al.

We should do a much better job explaining how our approach contributes to the literature. While others do study mixing real and generated data, our work pushes the field forward in important ways:

Reason #1: We frame our analysis from an extremely practical real-world engineering constraint: finite compute. Other papers study model-data feedback loops in ways that aren't clearly motivated by practical considerations. This framing is important.

Reason #2: Other papers add complications that make drawing clear insights difficult. For instance, Bertrand et al. 2023 studies iterative retraining, where each new model is initialized from the previous model's parameters and the each new model’s optimizer is initialized from the previous model's optimizer’s state This helps mitigate model collapse because each model is then limited in how far it moves from the preceding model, but this iterative retraining poorly matches real-world conditions as frontier models aren't (to the best of our knowledge) initialized from their predecessors' parameters.

Reason #3: The real+synthetic data loops of previous papers e.g., Bertrand and Alemohammad both focus on real data plus synthetic data drawn only from the most recent model. But in reality, if one is scraping the web, we know of no way to filter the collected data for data generated by the most recent generation of models. Our Accumulate-Subsample setting is meant to more faithfully model reality: synthetic data are added online, and are vacuumed up with real data to train the next model.

Reason #4: Briesch et al. 2023 is difficult to interpret because the results of their v1 and v2 Arxiv manuscripts (specifically, the key Figure 4) appear to reach slightly contradictory conclusions that the authors don't address.

In the revised manuscript, we will better contextualize our contributions relative to these prior works, explicitly highlight the practical relevance of our approach, and more clearly explain how our findings extend beyond what was previously known.

Definitions and Realistic Assumptions

I also want to note that the literature is not unanimous on the claim that accumulating data prevents model collapse. That is an ongoing discussion. (e.g. Dohmatob et al. B).

We agree that the field doesn't have a consensus. This is partially because the field has defined model collapse in multiple and sometimes conflicting ways (a point made by a recent position paper https://arxiv.org/abs/2503.03150) and partially because different papers make assumptions that are unrealistic.

This is why we feel it is so important to construct experiments in a way that emulates reality as faithfully as possible. In our opinion, the field has drawn conclusions from experimental setups that are oftentimes not well grounded in real-world considerations.

While we appreciate your review, you are mistaken about your claim that we primarily replicated experiments from other works, and the questions that you asked were largely answered in the paper. To explain line-by-line.

It seems like the manuscript just reproduces these experiments [from Bertrand, Alemohammad, and Shumailov] in a unified setting with some take messages.

This is false.

Shumailov: Shumailov proves that model collapse occurs for KDEs and Gaussian fitting in the REPLACE SETTING. A main point of our paper is that if you repeat these experiments IN THE ACCUMULATE SETTING, then collapse does not occur. We are the first to do this, and thereby directly compare and contrast the replace and accumulate settings.

Alemohammad: This paper again tests model collapse for generative image models in the replace setting. We don’t test generative image models at all in this paper, and we don’t use the replace setting.

Bertrand: Bertrand tests model collapse for language models in a variation of the accumulate setting. In this case, the proportion of real data is fixed but non-zero throughout all iterations. We allow the fraction of real data to go to 0 in our version of the accumulate setting, which marks a significant conceptual difference from Bertrand’s setup.

All experiments in this paper are original, except for the pretraining experiment, which we extend from Gerstgrasser et. al. to include more model-fitting iterations.

Overall, our Section 2 makes a novel and important contribution by identifying seemingly contradictory claims in the model collapse literature and comparing them in a head-to-head manner to resolve them.

The evidence in Sec 4 is not very rigorous to answer the key open question mentioned in the first paragraph of Sec 4.

We provide an F-test of whether the proportion or the cardinality of real data dictates model performance. This is a standard and fully rigorous statistical evaluation. The high statistical significance (p-values of $6.9e-25$ and $4.6e-25$ strongly supports our conclusions. If you disagree, please let us know why and what you would recommend instead.

It is unclear how to get the value of $1,024$ to support the claim that "when the number of real data is 1024 or lower, we find that there is a small but non-zero amount of synthetic data that improves the test loss when it is included."

This value is our result, shown in Figure 5, where the curves for real data samples ≤1024 show improvement with synthetic data addition, while curves for samples >1024 do not show this benefit. We are not making a theoretical claim about the number 1024; we are reporting the empirical result we observed across multiple experimental runs.

We believe these clarifications help address the reviewer's concerns and demonstrate both the novelty and rigor of our work. Additionally, the review does not discuss any strengths of the paper, and you left most of your review sections blank. We hope that you can provide more information about why you gave such a low score and also provide specific constructive criticism for how we can improve the manuscript.