How bad is training on synthetic data? A statistical analysis of language model collapse

We thank the reviewer for the positive feedback and suggestions. We recognize that our main statistical analysis results are based on the more restrictive recursive training setup and that our experiments are conducted on a small model. Due to limited computational resources, we did not conduct experiments on larger language models. In addition, we address the first limitation in App. C.2 (Fig 4) by performing experiments in more realistic settings “Most Recent Models” and “Randomly Sampled Data”. These two settings are meant to model the scenario of collecting text data online which have been contaminated with machine-generated content. In particular, when data generated by the most recent K models are sampled as the training set (the Most Recent Models setting), model collapse is delayed but still eventually happens, qualitatively similar to the Fully Synthetic setting. On the other hand, our theoretical result suggests that the Partially Synthetic setting behaves differently qualitatively. The deviation probability in Theorem 4 provides guidance on how much real data to incorporate to curb distribution shift.

We thank the reviewer for their feedback and suggestions. We provide responses to the reviewer's main concerns.

1. To study the impact (over multiple generations) of using synthetic data in the pretraining of LLMs, we consider recursive training where each generation is trained on either synthetic data or a combination of synthetic and real data. The Fully Synthetic setting (while it may not correspond to any practical use case) proves the point that information will eventually be lost over generations if we train new LLMs solely on synthetic data. On the contrary, in The Partially Synthetic setting (real+synthetic), degradation in model performance can be controlled if we carefully choose the ratio of real to synhetic data

2. Synthetic data can indeed be generated and used in various ways. E.g, one could prune the synthetic dataset to only keep low probability samples (to make them more present in the training data for next gen LLMs). In this paper, we only consider synthetic data obtained directly by sampling the output of an LLM (without post-sampling processing). Other approaches are beyond the scope of this work. We will include this discussion in the revised paper.

3. Our theoretical results consider a simple model for tractability. Extending the theory to transformers is currently challenging given the fact that the transformer architecture is highly complex. However, our experiments demonstrate the generalizability of our findings (derived on the simple model) in the context of transformers. In Fig 3, the Fully Synthetic and Partially Synthetic settings behave qualitatively differently – the former exhibits model collapse while the latter stabilizes in terms of model deviation. Even though the exact quantification of distribution drift is difficult to estimate for transformers, the result of this experiment aligns with our theoretical derivations in Theorems 1 and 4.

4. We respectfully disagree with the reviewer on this point. We do not provide experiments with larger models because it is computationally infeasible given our compute budget. In this regard, we would like to refer the reviewer to the last point of the Review Guidelines, where it is stated that "we ask that you take into account that most researchers do not have access to large-scale compute".

5. Extended limitations section: we will include the aforementioned points and concerns raised by the reviewers.

We hope our response clarifies some potential misunderstandings.

Given that the discussion deadline is in two days, we would appreciate it if the reviewer could provide feedback on our rebuttal, particularly noting if there is anything unclear.

We thank the reviewer for the constructive feedback and the recognition that our work is among the first to provide direct theoretical insights into recursively training language models. We would like to bring the following to the reviewer’s attention:

1. We choose to give a concise presentation in Sect 2.1 due to space constraint. The distinction between $x_l$ and $e_j$ arises from the assumption that contexts are represented by canonical (basis) vectors $e_j$ (i.e. one-hot embeddings in ML language), as explained in Sect 2.1 (“Note that in current …”). Additionally, in Eq (1), $M$ denotes the number of all (context, next-token) pairs in the dataset, while $N=|C_j|$ denotes the number of data points with context $j$ so $N=N_j$ in fact depends on $j$ and $M=\sum_j N_j$. We suppress the subscript of $N_j$ for brevity. A derivation for $p^{(m)}$ being unbiased will be added to the appendix.

2. We remark that our main results (in particular Theorems 1 and 4) hold context-wise and easily extend to all contexts under certain conditions. For example, if we assume that the number of synthetic samples $n_j$ for each context j is fixed across generations, we directly obtain uniform bounds on the expected collapse time or distribution shift which holds true for all contexts. A natural way to relax this assumption is to consider the autoregressive model governed by Eq (5) so that $n_j$ depends on the conditional distributions given other contexts, but this significantly complicates the analysis. Instead, in Sect 4 we perform exactly such an experiment with transformer model. In the top-left corner of Fig 3, we report the average deviation from the original distribution (across 300 randomly sampled contexts) which confirms the dichotomy between whether or not original data is retained for training, in accordance with our theoretical findings.

3. Our contributions beyond Shumailov et al. (2023) consist in the direct quantification of LM deterioration in recursive training. Their treatment on LM is indirect: theoretically, they only analyze Gaussian distributions which do not apply to LMs where output distributions are discrete. In contrast, our setup is more closely related to the actual setting in LM recursive training and we provide direct analysis accordingly.