Strong Model Collapse

We thank the reviewer for their positive reception of our work and their appreciation of the importance of the topic and the elegance of our approach. We will now address their questions and comments.

Could the authors detail more the key steps/ideas of the proofs in the main paper (paragraph Proving Theorem 1)? I believe such tools could be beneficial to the reader, even in other applications.

Sure. We have updated the manuscript with a modified version of that paragraph (see text in blue) which explains the key steps of the proof. Note that proofs of all results are given in the appendix. Also, there is a detailed ToC in the beginning of the appendix to help the reader navigate the proofs.

Could the authors discuss the estimation of the quality of synthetic data (parameter ) on the real applications with MNIST and BabiStories? How would one assess it when training a large-scale model like an LLM or a VLM?

For the quality of the MNIST and BabiStories datasets, we directly assess the accuracy or perplexity of the generator to evaluate the quality of the synthetic data, as also noted by Reviewer Wc8A. Additionally, we have included an ablation study in Appendix A.3 to examine how the quality of synthetic data impacts the scaling curves.

For general pretraining data for LLMs and VLMs, evaluating data quality is a broad and evolving area. Researchers have proposed various metrics in natural language processing and have utilized scores from pretrained models, such as the CLIP score [1], to evaluate image-caption pairs. Scaling these methods to assess large, general-purpose corpora remains an open challenge and an active area of research.

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

Could the authors discuss in more length the "cost" of approximating neural networks as random projections? In particular, what is lost in terms of generality, does it approximate any architecture (e.g., can LLMs be studied in the NTK or infinite-width limit?)

For the purpose of studying the impact of model size on the model collapse phenomenon, we choose to analyze ridge regression with random projections because this setup offers analytical tractability while producing a phenomenology which is strictly richer than in the case of classical linear models (with no random projections). Furthermore, still using free probability theory and the Gaussian equivalence theorem [Goldt et al. (2022)], we can easily extend our theory to allow for the nonlinearities and other complexities in the NTK regime, as in (Adlam & Pennington, 2020; Tripuraneni et al., 2021) in studying other phenomena like multiple descent and covariate shift, albeit at the cost of more complicated fixed-point equations describing the limiting value of the test error.

The question of whether NTK can approximate LLMs is broad with no simple answer. In any case, we can say the following (1) such an NTK would have to be finite-width, since infinite-width NTK's cannot account for feature learning (which is definitely present in LLMs). (2) In the so-called "lazy regime", such an approximation is plausible, due to the universality established in (Chizat et al., 2019).

Typos. We also thank the reviewer for the typos they have pointed out.

Paragraph Random Projection Models (L224 & L225 ): it seems that dimensions of $S$, $x$, $v$ do not match

Indeed, there is a mild typo (missing transpose on $S$): $Sx$ should be $S^\top x \in \mathbb R^m$, with $x \in \mathbb R^d$, $S \in \mathbb R^{d \times m}$, and $v \in \mathbb R^m$. This has been fixed (see blue text in updated manuscript).

We also thank the reviewer for the other typos they have pointed out (for example, the first "$P_1$" should indeed be "$P_2$" on L299; etc.). They have all been fixed.

We thank the reviewer for their positive reception of our work. We will now address their questions and comments.

I'm a bit unclear on the takeaway from the experimental results with GPT2. Specifically, the paper seems to make mixed claims for the case of large models beyond the interpolation threshold; it is said that large models may mitigate the collapse beyond the interpolation threshold and that large models tend to amplify collapse beyond the interpolation threshold in the experimental results.

The term "beyond the interpolation threshold" refers to increasing the network size beyond the point where a fixed dataset can be fully interpolated. In the GPT-2 experiment, however, we vary the number of tokens. As the number of tokens increases, the setting transitions from interpolating all data to not interpolating. Beyond the interpolation threshold corresponds to scenarios with fewer tokens, where a larger model mitigates collapse and achieves better performance. Conversely, when the parameter size is insufficient, as shown in the right end of Figure 5 (right), larger models tend to amplify collapse.

For the empirical results, it is said that the curves are expected to plateau in Fig 5. Can this be shown empirically by further scaling?

Theoretical predictions suggest that the scaling curves will plateau. In the empirical setting, with the x-axis now on a logarithmic scale, extending the range slightly already doubles the already high training requirements of a pretraining task, making the demands even more substantial. Moreover, to properly demonstrate scaling law curves for a long range, it would be necessary to further increase the model size when the training token count increases, significantly raising computational costs. Unfortunately, these experiments are too expensive and resource-intensive for us to conduct as part of our rebuttal (it would take more than 2 weeks running on 8 V100s).

Our theory predicts that optimal performance is bounded by a constant that depends on $c^2$ (data quality) and $p_2$, the proportion of synthetic data. The case of small $p_1$ is already demonstrated in Figure 5 (Right), and we expect similar trends to hold when extrapolating the other curves in Figure 5 (Left).

Can the double descent phenomenon be showed more clearly in the experimental results?

We clearly demonstrate the double descent curve in Figure 8 in Appendix A.2 using the MNIST dataset. As the number of samples increases, the curves exhibit a characteristic "down-and-up" pattern. Notably, when the sample size is fixed, the performance rankings of models with different parameter sizes change depending on the sample size, further illustrating the double descent phenomenon.

In the current Figures 10 and 11, provided in response to Reviewer Wc8A, we observe the double descent behavior more explicitly when increasing the model size while keeping the sample size fixed. This closely mirrors the theoretical double descent curve shown in Figure 4. With different combinations of the number of synthetic data $n_2$ and hidden dimension $d$, the figure captures various segments of the double descent curve.

In Figure 5, the double descent behavior can also be interpreted through the changing performance rankings of models with different sizes as the sample size varies.

Separately, I would be interested in more analysis specifically in the high quality data setting, ie what can be said about the amount of model collapse as the synthetic distribution approaches the real distribution?

In our theoretical framework, high-quality synthetic data corresponds to small values of the quality $c^2$ parameter (smaller is better; refer to Definition 1). The amount of model collapse is a decreasing function of $c^2$. Letting $c^2$ tend to zero (i.e synthetic data with quality tending to perfection) in Theorem 1 and Theorem 2 and their corollaries (e.g Corollary 1 of Theorem 1) recovers classical results. Of course, this limit also matches $p_2 \to 0^+$, corresponding to the setup where all training data is from the true data distribution. Our results show that for fixed positive values of $c^2$ and $p_2$ however small, model collapse persists. This is what we refer to as strong model collapse.

We hope that we addressed all your concerns, and that you are satisfied with our responses and changes. If so, please consider raising your score.

We thank the reviewer for their time and effort, and their positive appreciation of our theoretical contributions and illustrative experiments. We will now address their questions and comments.

Model Assumptions: I am unsure of the modelling of synthetic data and how well it translates into practice. Specifically, the authors model synthetic data using a label shift, assuming that the data (X) marginal remains the same. However, it seems unrealistic for autoregressive training (a key experiment in the paper), where the input tokens for next token generation come from the synthetic distribution.

Label-shift and covariate shift. At the cost of much more cumbersome theorems and formulae, theoretical results in our paper can be easily extended to accommodate scenarios where the covariance matrix of the synthetic data distribution $\Sigma_2$ is different from the covariance matrix of the true data distribution $\Sigma_2$, alongside the label-shift setup we used in our work. It is a matter of invoking the general bias-variance decompositions established in Appendix I and combining with the deterministic equivalents provided in Appendix E. We chose to keep things simple.

Let us note that an assumption like $\Sigma_1 \ne \Sigma_2$ alone (i.e without label-shift) corresponds to covariate-shift, a setup studied extensively in the literature Shimodaira (2000); Tripuraneni et al., 2021; (also see the classical work [1] listed below), but is not sufficient to capture model collapse as understood in the literature. Our work focuses on label-shift by choice. Let us recall here that the main payload of our work are negative results which indicate that the model collapse phenomenon persists even when synthetic and real data are mixed, showing that the findings of Shumailov et al. (2024); Alemohammad et al. (2023); Bertrand et al. (2023); Dohmatob et al. (2024a;b) are worse than anticipated. In line with prior work, the setup of label-shift is sufficient for highlighting our striking result, while keeping theorems as simple as possible. We will make these points clearer in the manuscript.

[1] Shai Ben-David et al. "Impossibility Theorems for Domain Adaptation", AISTATS, 2010.

Auto-regressive models. We adopt the setting of X-Y pairs theoretically, as it encompasses a broad application range in language modeling, particularly in post-training scenarios where the model receives a prompt (X) and generates a response (Y). This setup includes instruction tuning, standard fine-tuning, and tasks such as question answering and code generation.

We deliberately chose the GPT-2 autoregressive experiment because it is a pretraining tasks, where model collapse is claimed to have a significant impact (Shumailov et al. (2024); Dohmatob et al. (2024a)), rather than fine-tuning. Many related works conduct experiments exclusively on fine-tuning outcomes. In the current story dataset, prompts and responses also play a role in dataset curation: a set of prompts was first created and then used to query GPT-4 (Mistral-8x7B) and our trained GPT-2 to generate BabyStories (TinyStories) and the synthetic data. The dataset is subsequently utilized in the autoregressive pretraining process. The experimental results suggest that insights derived from the prompt-response setting can effectively generalize to pretraining tasks.

I understand that it might not be computationally feasible for GPT-2 in the given timeframe, but could the authors provide an ablation on the MNIST experiment with regards to the quality of synthetic data? That is, models trained on labels taken from increasingly worse synthetic classifier and the corresponding trends.

We have included the results of additional experiments on MNIST in Figures 10 and 11 in Appendix A.3. These figures complement Figure 8. In these experiments, we examine how the quality of synthetic data influences the relationship between the number of synthetic data points, model size, and performance. The quality of synthetic data is evaluated using the MSE loss of the generator on the test set, consistent with our approach of framing the classification problem as a regression task. For the same dataset and trained model, we also use test accuracy to evaluate the quality and performance in Figure 11. We also keep the y-axis in the same scale.

Final Note. The perceived contradictions in conclusions primarily arise from differing definitions of what constitutes avoiding model collapse, as detailed in our response to point 1 above. These differences have no bearing on the underlying modelling assumptions (as claimed by the commenter), but rather arises from a misguided view of the model collapse phenomenon in their own paper Gerstgrasser et al. (2024). The perspective and findings of our work adhere to the prevalent definition of model collapse in the field and align with practical considerations. We believe this perspective reflects the most realistic scenario for practitioners faced with training corpora pulluted with AI-generated content, as detailed in Point 1, Point 2, and Point 3 above.

Point 1. Concerning Gerstgrasser et al. (2024)

Scientific Contradictions with Prior Work ...

This statement from the commenter is inaccurate and misleading. Indeed the commenter's own work Gerstgrasser et al. (2024) (mentioned in their comment) doesn't avoid model collapse in any meaningful way. Let us explain.

In our introduction, we clearly define model collapse as "a critical degradation in the performance of AI models." Avoiding model collapse would mean to close the performance gap between training with real and synthetic data.

The definition adopted in the commenter's own work Gerstgrasser et al. (2024) describes avoiding model collapse as preventing recursive degradation when models are trained multiple times. The paper acknowledges that, while performance degradation is bounded when samples accumulate, there is still a performance loss. This definition represents only a partial condition for closing the gap between real and synthetic data. In this sense, Gerstgrasser et al. (2024) doesn't solve or mitigate model collapse.

In the broader literature on model collapse, the prevailing trend considers closing the performance gap as the primary criterion for avoiding model collapse. We chose to align with this widely adopted approach.

From a practitioner's perspective, closing the gap between real and synthetic data is a more actionable and relevant notion of avoiding model collapse. Simply ensuring non-diverging performance can still result in models failing to match the synthetic data generator's quality, thereby causing synthetic data to harm performance. Only when the gap is fully closed can the harm from synthetic data be entirely mitigated. We believe this principle should guide practitioners.

Due to the differences in definitions and the rationale for considering closing the gap as the correct definition, we have reviewed all related work on understanding model collapse and the strategies to avoid it through this widely accepted perspective.

Additionally, the referenced paper Gerstgrasser et al. (2024) was published at COLM (October 2024), after the ICLR submission deadline, with its camera-ready deadline in August. According to the ICLR author guidelines, papers published after July 1st are considered contemporary work.

Prior interaction with the authors of Gerstgrasser et al. (2024)

Unfortunately, one of the statements of the comment poster seems to break anonymity of this review process. We did have an interaction with the authors of Gerstgrasser et al. (2024) months ago when their paper first appeared on ArXiv. Let us succinctly summarize the conclusions of the said interaction:

• Technically insufficient. The technical contribution of Gerstgrasser et al. (2024) was too incremental and very weak. It was nothing more than a slight modification of one of the setups and arguments already presented in an earlier version of Dohmatob et al. (2024a) "Model Collapse Demystified: ...". Their results can be recovered as a trivial corollary of Theorem 4.1 of Dohmatob et al. (2024a). Also see Remark 4.2 of Dohmatob et al. (2024a).
• Misleading/inaccurate conclusions. The paper simply didn't solve model collapse in any reasonable sense (see our previous note on definition, further above, and also the related work sections of this ICLR submission), contrary to the claims made by Gerstgrasser et al. (2024).

Unfortunately, later versions of the paper Gerstgrasser et al. (2024) essentially ignored the constructive criticisms we kindly shared during this interaction, relegating our remarks to a footnote at the end of the paper. Therefore in its current state, we still don't think Gerstgrasser et al. (2024) is scientifically sound (for example, in terms of the technical contributions and most importantly, the misleading/inaccurate claims of solving/mitigating model collapse, as explained above), and therefore didn't feel the need to cite.

Point 2. Related Work on Avoiding Model Collapse

The statement from the commenter regarding "simple adjustments such as data weighting unless these strategies asymptotically remove all but a vanishing proportion of synthetic data from the training process" is a faithful interpretation of our theoretical results. To the best of our knowledge, this characterization does not contradict related works in the field with the same definition of avoiding model collapse. The claim is firmly grounded in theory, and our theoretical framework has been rigorously vetted by reviewers. Specifically, we examine Strategic Data Mixing with data source coefficients for data mixing and demonstrate that it will not prevent model collapse unless the strategic weights of synthetic data asymptotically vanish.

Point 3. Reuse of Experimental Methodology

Our choice to use GPT-2 and the BabiStories setting is driven by several factors: its role as a pretraining task, its ability to generate more 'real' data to support the scaling analysis shown in Figure 5 (Left), and the availability of comprehensive resources, including generation code, checkpoints, and training code, provided in [1]. These resources, combined with the use of Mistral to generate additional data for BabiStories, enabled us to successfully conduct the large-scale scaling experiments detailed in Figure 5.

The trained GPT-2 checkpoints, generated samples, training code, and generation code were obtained from the public GitHub repository of the Memory Mosaics paper [1]. We extended this codebase to perform our experiments and are fully prepared to provide evidence of this to the AC and reviewers if needed.

Furthermore, we emphasize that TinyStories and BabiStories have become widely recognized as standard dataset for affordable pretraining [1]. These datasets are valued for their efficiency in facilitating the pretraining of high-performing language models at moderate scales, as evidenced by numerous studies [2–10]. We thus emphatically refute the claim that we have used experiments from Gerstgrasser et al without acknowledgement - or else the same is true for Refs [1-10] below.

[1] Zhang, Jianyu, et al. "Memory Mosaics." arXiv preprint arXiv:2405.06394 (2024).

[2] Raises, Lying Leg, and Side Plank. "How to Generate and Use Synthetic Data for Finetuning."

[3] Cuervo, Santiago, and Ricard Marxer. "Scaling Properties of Speech Language Models." arXiv preprint arXiv:2404.00685 (2024).

[4] Radhakrishnan, Adityanarayanan, et al. "Mechanism for feature learning in neural networks and backpropagation-free machine learning models." Science 383.6690 (2024): 1461-1467.

[5] Gu, Yuxian, et al. "Towards Optimal Learning of Language Models." arXiv preprint arXiv:2402.17759 (2024).

[6] Shah, Harshay, Andrew Ilyas, and Aleksander Madry. "Decomposing and editing predictions by modeling model computation." arXiv preprint arXiv:2404.11534 (2024).

[7] Razzhigaev, Anton, et al. "Your Transformer is Secretly Linear." arXiv preprint arXiv:2405.12250 (2024).

[8] Sharifnassab, Arsalan, et al. "Soft Preference Optimization: Aligning Language Models to Expert Distributions." arXiv preprint arXiv:2405.00747 (2024).

[9] Zhao, Yize, et al. "Implicit geometry of next-token prediction: From language sparsity patterns to model representations." arXiv preprint arXiv:2408.15417 (2024).

[10] Zhong, Ziqian, and Jacob Andreas. "Algorithmic Capabilities of Random Transformers." arXiv preprint arXiv:2410.04368 (2024).

As the AC of the paper I clarify that my recommendation was to reject the paper. The decision was overturned to an Accept by the Program Chairs.