Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

Dear reviewer TmgS,

Thank you for your review.

is it possible to accumulate the real data as well? In nature the real data will be generated by humans over time. The experiments will provide more insights.

We agree this is an interesting question and a more realistic modeling assumption. For the current paper, we intentionally wanted to present a very pessimistic scenario of accumulating data, to show that even under pessimistic conditions, simply keeping previous data avoids model collapse.

In practical terms, for experiments accumulating real data does present a challenge, as you can’t create new real data in an experiment, so one would have to start out with just a fraction of the dataset in the initial iteration.

A clarification (as well as a detailed section of related work) from the authors will be appreciated.

We will update the manuscript with an in-depth 2-page Related Work section

To your specific question, our setting is different from the synthetic augmentation loop in Alemohammad et al! In their work, they primarily consider the case where a constant amount of synthetic data is replaced in each iteration, in addition to a constant amount of real data. That is, their setting is closer to what we consider the “replacement” setting, and they do conclude that this does not avoid model collapse. They do briefly look at accumulating data as well, but only in one part of one experiment, and they conclude that this merely slows model collapse down.

We will include a new appendix section closely examining Figure 7 of Alemohammed et al. We believe that a closer examination suggests that test error plateaus (and this would likely have been more apparent in subsequent iterations), consistent with our results and in contrast with their conclusion that “fixed real training data only delays the inevitable degradation of the quality or diversity of the generative models over generations."

Dear reviewer TmgS, Thank you so much for your prompt response to our rebuttal. You state we've addressed your concerns “to a certain extent”. May we ask what concerns remain? We would of course like to address them in full.

We thank Reviewer 7uQd for their review and interesting discussion. Our responses are below:

The exploration on larger model is desired [...] such as those with billion-parameter scales

We agree larger models would be better but billion-parameter models are computationally unaffordable for us. We pre-train from scratch, repeatedly, on growing data. 5 model-fitting iterations is equivalent to 20 full pre-training runs; the experiments in the paper together are well into the hundreds of full pre-training runs. To evaluate 7B parameter models on a 2T token starting dataset, we would need to train for 40T tokens for a single 5-iteration experiment – more than twice as many as Llama3! And this is not even taking into account generating the synthetic tokens. We cannot afford this.

Because of this, we carefully choose the TinyStories dataset, as this has been designed specifically to allow for meaningful experimentation with smaller models.

That said, if you have a specific suggestion of a model size and dataset that are both achievable with reasonable compute budgets, and would provide insight that our current experiments do not, we would be very happy to discuss this!

The paper mainly relies on linear models for its theoretical analysis [...] this simplification might not fully capture the complexities of real-world scenarios.

We agree that linear models may fail to capture real-world scenarios, which is precisely why we complemented the theory with three medium-to-large-scale realistic experiments on deep generative models across 3 data modalities.

Our main results are these extensive experiments. We will swap the order of experiments & theory to emphasize this.

This paper lacks of discussion and comparison with other popular methods in this area

Could the reviewer please clarify what “other popular methods in the area” means? Our work is scientifically studying a particular system, i.e., sequences of generative models trained on outputs from their predecessors.

The discussion about the related work is missing.

We have written a 2 page Related Work and will add this to the Appendix.

Dear Reviewer 7uQd, thank you again for your review and comments. We wanted to follow up to check if we’ve addressed your questions and concerns? If not, we’d be keen on discussing further in the remaining time.

Thank you for your in-depth review and the interesting and on-point discussions you raise.

Lack of discuss on data quality [...] It is essential to engage with the leading edge of synthetic data in each domain to advance our understanding.

There is growing evidence suggesting that language models are capable of generating synthetic data of superior quality compared to real data [1], which can be more beneficial for certain downstream tasks.

We agree that research on using synthetic data to improve models or accelerate learning is an exciting new topic.

However, we see this as a step beyond our paper. Our paper is not asking the best way to use synthetic data. Our paper is asking whether future generative models will become useless in a pessimistic future where synthetic data is dumped without restraint on the internet. Our conclusion is no. Our point is not that random synthetic data is good for models, but that such data is not especially harmful so long as data accumulates. We believe this is an important and timely point to make, considering model collapse has recently been picked up by mainstream media as a catastrophic problem that threatens the future of generative AI.

We add to our Discussion that identifying how to best use synthetic data is an exciting research frontier and cite your recommended paper “Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling”.

Impact of data size in comparisons

Appendix C includes this ablation (“Controlling for dataset size”) and our updated manuscript will have more experiments. For hyperparameters, is there any specific experiment you would like to see?

Test loss on real data might not be the best metric to compare

Test loss on real data is an extremely standard metric for evaluating models. While today’s language models indeed have significant post-processing applied, such methods rely on the base models having low pretraining loss. Applying DPO or some other preference optimization method to a high-perplexity LM will not work.

I would suggest the author measure the loss on a held-out dataset

To be clear, we do measure the loss on a held-out split of the real data.

maybe loss on synthetic data is a better metric to compare

Measuring the model’s loss using model-generated synthetic data opens the door to undesirable minima. For instance, if the model always outputs the same token, its cross entropy loss on the generated data will be 0, but its outputs are useless.

Dear Reviewer Vd4L, thank you again for your review and comments. We wanted to follow up to check if we’ve addressed your questions and concerns? If not, we’d be keen on discussing further in the remaining time.

Thank you so much for your response! I agree with you that working in a setting where the generated data's quality is inferior to real data is still valid and interesting to show. However, I would encourage the authors to discuss the following points in their revisions:

• Would the observed phenomenon still hold if the quality of synthetic data exceeds that of real data (which is of mixed quality)? This is particularly relevant for GPT-generated texts, where it has been shown that training on synthetic data can be more beneficial than training on real data. I believe this discussion is crucial, as we have already observed this empirically.
• Why do different patterns exist between language, molecular modeling, and image synthesis during accumulative training? How do these modalities differ in nature?
• Regarding dataset size, my point was that the comparison between replacement and accumulation differs in the number of optimization steps due to varying dataset sizes. Controlling the dataset size to be the same would be more helpful in identifying how the distribution of data composition affects model performance. I believe the conclusion will largely hold for the current setting though.

I have raised my score to 6, and I am happy to have further discussions.

Dear Reviewer Djpz,

Thank you for your review and useful comments. Overall, we want to stress that the main message of our paper is to show that accumulating vs replacing data leads to a stark difference in model collapse (or lack thereof), a point overlooked in prior literature. We believe, as you also state, that we thoroughly demonstrate this. To your specific points:

I feel like the major weakness is the limited scope of theoretical analysis to linear regression [...] the conclusions may not immediately generalize to more complex models.

We consider the main contribution of our paper to be the experiments, which are more extensive and diverse than prior work in the model collapse literature. The theory was intended as the simplest analytically-tractable example to explain mathematically why accumulating data makes a difference. In our revised manuscript, we swap the order of experiments and theory to emphasize the experiments.

the authors could expand the theoretical framework to include non-linear models.

Our results hold for nonlinear kernel regressors; we would be happy to state such results in the appendix. However, deriving results for other nonlinear models like logistic regression is non-trivial, as we know of no closed-form solution. In general, theory in this direction is very hard. We also feel that this would detract from the manuscript’s main message, which we believe is an important and highly timely one, and one for which we provide ample evidence.

Plus, authors can validate the practical applicability of their claims on larger model settings,

We want to clarify that we pre-train these models from scratch, at each iteration, with growing dataset sizes. Thus, to evaluate a sequence of models with just 5 model-fitting iterations, we need to train for the equivalent of 20 full pre-training runs (1+2+3+4+5 for accumulate, 1+1+1+1+1 for replace). The experiments in the paper altogether constitute hundreds of full pre-training runs. Doing this on billion-parameter models and web-scale data would require compute matching OpenAI, Google or Meta, which sadly we do not have.

Because of this, we carefully chose the tinystories dataset precisely because it was specifically designed to allow insightful experiments with smaller models. In particular, it was designed to be well-fit by 10-30M parameter models, and we evaluate up to an order of magnitude larger than that.

Dear Reviewer Djpz, thank you again for your review and comments. We wanted to follow up to check if we’ve addressed your questions and concerns? If not, we’d be keen on discussing further in the remaining time.

Dear Reviewer, as the discussion period is coming to a close soon, we wanted to follow up once more to check if we have addressed your concerns? We would be keen to fully utilize the remaining time to discuss further if any questions or concerns remain.

(continued from Part 1 above)

However, even if one did increase the amount of synthetic data with iteration count, Theorem 3.2 coupled with Corollary 3.3 in Dohmatob et al. (2024b) would tell us that the amount of real data was all that mattered – if the amount of real data is large, we overcome model collapse. If one only had synthetic data (and no real data), no matter how large, it would be impossible to regain the origina real-data scaling laws. The scenario we study is highly inspired by these pioneering works, but still, in our view, different. We consider the case when we keep augmenting synthetic data (generated by the previous model trained on all the previous data so far) as iterations progress, much akin to how – in our view – the internet evolves. We observe that we can avoid model collapse in this setting. The analysis of previous models in our case is more involved, since the data used for training at iteration n is not homogeneous – different models from the past impart different statistical aspects to different parts of the training data. We also note a related augmentation model studied by Jain et al. (2024) – they perform risk minimization augmenting real data with synthetic data available from a potentially different independent source. One of their messages is that augmentation of (even) pure noise can be looked upon as adding a ridge penalty and hence, in certain cases, can improve test error. Their setup, however, is different from ours, since the synthetic data in their setup is not obtained by a learning algorithm employed on the real data, and the process is not iterative. However, morally, each iteration of ours involves risk minimization on data statistically composed of an equal mixture of data generated from the previous models, and hence each iteration of ours can be mapped to the general framework developed in Jain et al. (2024), although the dependencies among the various models trained in our setup introduce theoretical complications that do not seem to be too easily addressed by the theory developed in Jain et al. (2024). Shortly after v1 of our manuscript was uploaded to ArXiv, two other manuscripts appeared, dealing with the theoretical aspects in a setting similar to ours. Theorem 1 of Marchi et al. (2024) obtains the same square summability scaling of the variance as us. Seddik et al. (2024) studies collapse in language models in both purely synthetic and partly synthetic regimes and obtains deviation bounds as model iterations progress.

Considering Accumulating Data The two papers we found that partially considered accu- mulating data are Mart´ınez et al. (2023a) and Alemohammad et al. (2023). Alemohammad et al. (2023) did so in one-half of one experiment: StyleGAN2 trained on FliqrFaces 128×128 (App. Fig. 8). The authors concluded that accumulating data does not avoid model collapse, but merely slows it down. However, we believe that a closer examination of their results (App. Fig. 8) reveals that accumulating data causes the test error to plateau to a relatively low error with increasing numbers of model-fitting iterations. This result would support our conclusion that accumulating data avoids model collapse and does not merely delay it. The results from Mart´ınez et al. (2023a) are harder to evaluate; model collapse only seems to occur when the amount of synthetic data added per model-fitting iteration is 2× the total amount of accumulated data, and the subsequent work by the authors switched from accumulating data to replacing data (Mart´ınez et al., 2023b). We think understanding what conditions and why these discrepancies exist is an interesting future direction. Avoiding Model Collapse Several papers present methods for avoiding or slowing model collapse. Bertrand et al. (2023) shows in the replacing data setting that model collapse will not occur if the initial generative models approximate the data distribution well enough and the proportion of real data is sufficiently large with respect to the synthetic data. Dohmatob et al. (2024b) similarly demonstrates that in the replacing data setting, carefully selecting real data to mix with synthetic data can avoid model collapse. Other solutions may also be possible in various models and under various assumptions. To our knowledge, no paper has claimed an “optimal” strategy to avoid model collapse, and neither has ours.