Model Collapse Demystified: The Case of Regression

Thank you for your thoughtful review and the dedicated time and effort you invested. We genuinely appreciate your recognition of the key strengths in our work.

1. To begin with, the model considered here is extremely simplistic and places itself much closer to the distillation literature, rather than generative literature ...

Thanks for the question. We have addressed it in the general response, and will include a discussion comparing our results with previous theoretical results in the paper.

2. The particularly interesting section establishing the scaling laws, which goes significantly beyond the model in Shumailov et al. seems to be only briefly discussed, and the novel insights from this analysis are not discussed beyond the paragraph in the introduction.

The purpose of Section 4 to which the reviewer is referring is to quantifiable show how model collapse manifests as a change of scaling laws, in a setting which is close to neural networks (in certain regimes) and large language models. We recall that scaling laws (Kaplan et al. [26] Hoffmann et al. [24]), which relate the test error of a model to the sample size, model size, and amount of compute used, are one of the main instruments used by practitioners in strategic allocation of resources in the design and implementation of large language models (ChatGPT, Llama, etc.). Indeed, the results of Section 4 predict that the scaling laws w.r.t dataset slow down (i.e smaller exponents) in the presence of synthetic data: a much larger sample size might be required to achieve the same decrease in test error as with real data. Moreover, the test error for models trained on synthetic data will eventually plateau as a function of sample size, in sharp contrast to the idealized classical picture [24, 26] where the test error for models trained on real data is predicted to decrease forever. These theoretical results are illustrated in Figure 1(b) and Figure 4.

We will add the discussion to the paper.

3. While it may be interesting as a generic model, the potential insights it might bring into the process of model collapse are much more important, and for this it needs to be put into context of the limitations of previous proposed models of the phenomenon.

We have addressed the point raise here in the general response and will include it in the paper.

4. The kernel ridge regression model.

Thank you for the question. The main difference between linear regression and kernel regression lies in the use of the kernel feature mapping. In our setting, generalizing from linear to kernel regression is straightforward. We chose to use linear regression because it is simpler in terms of both notation and understanding. As noted at line 47, "we present the linear regression setting for simplicity; the generalization to the kernel setting is straightforward." and we have discussed it in more details in Appendix B. We will also add a footnote at line 30.

5. The connection to self-distillation.

Thank you for raising this point, which caused us some grief to formulate succintly in the paper. Allow us to expand here, to address your very important question (Model collapse versus self-distillation): Our setting is inspired by the model collapse phenomenon, where more and more there will be vast amounts of synthetic data generated by users posted online, which will necessarily enter the training set of the next foundation model. In this case, we do not have the groud truth label, and the generation of synthetic data is not controlled by us, but by other users. Therefore, we adopt the setting with only synthetic labels with added noise.

Specifically, in our setup, at generation $n>0$, one doesn't have access to the true labels $Y_0=f_0(X) + noise$ for the training samples $X$, but rather to some $\hat Y_n = \hat f_n(X) + noise$, where $\hat f_n$ is an unknown function, which synthesizes fake labels iteratively; the integer $n$ is the number of iterations. $f_0$ is the ground-truth labelling function (unknown). In our work, we make the structural assumption that $\hat f_n$ is obtained by iterative / successive regressions on a true dataset $D_0=(X_0,Y_0)$, and then new noise is injected every single generation. The fake labels $\hat Y_n$ are like labels for images $X$ on internet, or outputs from a LLM like ChatGPT. One doesn't have any control over the creation of these labels, especially because of the noise injected at each stage. In practice, one doesn't even know that they're not real labels. The final model is learnt on the dataset $(X,\widehat Y_n)$.

In the self-distillation setting, one has access to the true labels $Y$. One decides to replace $Y$ with some $Y_n := F_n(X,Y)$. Here, $f_0$ is the ground-truth labelling function (i.e $f_0(x) := x^\top w_0$ in the case of linear models). The final model learned is via regression on the dataset $(X,Y_n)$. It is up to the practitioner to chose what the number of iterations $n$, and in fact Mobahi et al. showed that there is an optimal value of $n$ depending on the distribution of the data and the model class over which the regression is done. This is what we mean by having control over the data synthesization process. Thus, mathematically, the difference boils down to the absence of fresh noise when data is regenerated in SD, which leads to completely different behavior to our setting, where we do not control the noise injected at each generation. Allow us to use a metaphore here. Model collapse is like the rain, while SD is like the shower. The latter can be controlled by the user (for example, turned on / off), while the user has to cope with the former.

6. Section 4 ends very abruptly, and no in-depth discussion of the model is presented.

Thank you for the question. We agree that additional explanation is necessary. We have provided a discussion in response to question 2 and will include it in the paper.

Please allow us to use the official comment section to address the further comments you have in your review.

7. line 30 - kernel regression is not analysed in this paper

We have addressed the point raised here in the response to question 4.

8. line 47 - you do not mention what \epsilon is

Thanks. $\epsilon$ is the label noise, and we will add it.

9. line 54 - Insights from this formula needs to be related to existing mathematical models of model collapse.

We have addressed this point in the general response by comparing our theory with previous ones. We will also connect it with the results from Shumailov et al., as their work is the most similar to ours in mathematical form.

10. line 118 - $\hat{\omega}$ is a vector input to $E_{test}$? Or is it a random variable for expectation?

Sorry for the confusion. $\hat{w}$ is the input and there is no expectation over it. The notation $E_{test}(\hat w)$ stands for test error of the downstream model $\hat w$ as defined in equation (3).

11. It does not seem to be clarified what the effect of sampling different design matrices is.

In the "Dependent Case" at line 247, we analyze the scenario where the same design matrix is used for all generations, i.e $X_m=X_0$ for all $m<n$. Further restricting to the case $T_m=T_0 < d$ for all $m<n$ (i.e over-parametrized generator), our Theorem 3.5 shows that in this case, even in the absence of any label noise, there is model collapse due to an increase in the bias term in the test error. This increase in proportional to $1-\eta_0$, where $\eta_0:=T_0/d<1$. In the "Independent Case" on line 257, we study the scenario where an independent design matrix $X_m$ with $T_m<d$ rows is resampled at each stage $m<n$ of the generation process. In this case, our Theorem 3.6 shows that increase in bias is proportional to $1-\eta_0\eta_1\ldots\eta_{n-1}$, which is typically much larger than $1-\eta_0$.

12. around line 128 - notation seems a bit overloaded, with n subscripts being dropped, even though you may not have the exact same variables across iterations? E.g. X_n can not always be denoted by X?

In our theory, we focus solely on the last generation, as illustrated by the purple boxes in Figure 3. Since we fix all previous generations and examine only the last one, it is consistent to denote $X_n$ with $X$.

13. line 155 - the whole paragraph seems to be describing almost the same result as Shumailov et al., but does not mention it.

We will acknowledge Shumailov et al. as we arrive at a similar expression depending on the number of samples across all generations. However, our discussion focuses on how to analytically mitigate model collapse by using more samples during generation and whether this is an effective method.

14. line 173 missing citation

It is intended to be a question mark. We have removed the question mark for better understanding.

15. I am slightly unclear why section 3.4 is underparametrized - could you clarify?

As stated in the beginning the section, this refers to the scenario where each stage of the recursive synthetic data generator is based on at most as many samples as input dimensions, i.e $T_0 \ge d$. In this scenario, there are more data points than dimensions, resulting in a unique solution for the regression. Our theory reveals that in this setup, model collapse is due to an increase in the variance of the downstream estimator. Conversely, when $T_0 < d$, we refer to the synthetic data generator as over-parameterized, as there is no unique solution to the regression problem without regularization. In such cases, various interpolators can achieve training error: OLS produces the one with minmal Euclidean norm. As we see in Section 3.5, model collapse appears in this scenario even in the abscence of label noise

We thank the reviewers for all their questions and hope we have addressed them thoroughly.

We thank the reviewer for their further comments.

• About kernel regression

We agree with the reviewer that the statement about kernel regression should be a remark that our analysis can be straightforwardly extended to the the case of kernel regression. This will be rectified.

• About self-distillation

We would like to stress that our setup cannot be reduced to a noisy version of self-distillation. Let us explain. Self-distillation for linear regression would amount to a very special instance of our analysis where (1) $X_0=X_1=\ldots=X_{n-1}=X_n=X$ and (2) $\sigma_0=\ldots=\sigma_{n-1}=0$. That is, there is exactly one design matrix which is used in the data generation process and in the downstream estimator, and also no additional source of label noise is present at the end of each generation.

In the general setup considered in our work, (1) is not imposed. We typically assume that $X_0,X_1,\ldots,X_{n-1},X_n$ with $X_n=X$, are all independent random matrices. An exception is line 247 ("The Dependent Case") of Section 3.5, where we assume $X_m=X_0$ for all $m \le n-1$, and independent of $X_n=X$. That setup (considered for the purposes of showing that model collapse can still occur in the absence of label noise) also assumes $\sigma_m=0$ for all $m$; the analytic picture which emerges (Theorem 3.5) is already drastically different from what one would get from self-distillation (corresponding to additional assumption that $X=X_0$).

Thus, not only are the processes in self-distillation and model collapse different, the theoretical models are drastically different.

We will add this discussion in the main paper.

I am not sure we are reading the same paper, which result are you referring to?

By "an optimal value of $n$" we refer to the number of distillation steps that yields the best performance, as discussed in Mobahi et al. (ref [36] of our manuscript). Specifically, the authors boldface highlight it in the introduction (specifically, the contributions paragraph) like so "This implies that a few rounds of self-distillation may reduce over-fitting, further rounds may lead to under-fitting and thus worse performance". This is accordance with the theory developed in their paper, and is empirically confirmed in their Figure 3.

[36] Hossein Mobahi et al. "Self-distillation amplifies regularization in hilbert space." Advances in Neural Information Processing Systems 33 (2020)

We appreciate your review and positive evaluation of our work, as well as your acknowledgment of its strengths.

1. The analysis primarily focuses on regression tasks, which may not fully capture the complexities of more advanced machine learning models like large language models or image generators. The theoretical framework relies on certain simplifying assumptions (e.g., Gaussian data distribution, linear models) that may not always hold in real-world scenarios. While the paper includes experiments, they are primarily on synthetic datasets. Validation on real-world, large-scale datasets could further strengthen the findings.

We addressed it in the general response.

2. While you propose adaptive regularization as a potential solution to mitigate model collapse, could you elaborate on other practical strategies that might be effective? For instance, how might techniques like data filtering, model ensembling, or continual learning potentially address the issues you've identified?

Scaling laws tend to deteriorate when incorporating synthetic data. As demonstrated in our Theorem 4.1, adding more synthetic data can either break scaling laws or significantly worsen them. To achieve improvements in the next generation of foundational models, it is crucial to preserve these scaling laws. Data filtering is certainly beneficial in this regard. Additionally, watermarking synthetic data is becoming increasingly important as it aids in distinguishing high-quality real data from synthetic counterparts.

Adaptive regularization in Corollary 4.2, akin to early stopping, suggests that early intervention during training can prevent model collapse. Furthermore, using the current model to filter and bootstrap the data could be a viable solution. This approach can help maintain the integrity of the training process by ensuring that only the most relevant and high-quality data, whether real or synthetic, is used.

We will add the discussion to the paper.

3. Given your findings on how model performance can degrade with synthetic data, what implications do you see for AI safety and robustness, especially as AI-generated content becomes more prevalent on the internet? How might your work inform strategies for maintaining model quality and reliability in an ecosystem increasingly populated by AI-generated data?

The key takeaway is that scaling alone is insufficient when synthetic data is involved; these data can alter traditional training paradigms, particularly as optimal regularizations change. Consequently, new algorithms with appropriate regularizations are necessary. Maintaining the quality and reliability of models requires rigorous validation processes focused on data quality. More effort must be directed toward data curation to ensure the integrity of training sets. Additionally, developing tools and methodologies for detecting AI-generated content, such as watermarking, is essential to distinguish synthetic data from real data and to maintain model reliability.

We remain available to address any further questions you may have. We hope our responses address your questions and encourage you to consider raising your evaluation score. Thank you.

Thank you for your feedback, especially on the presentation of our material, which is challenging, given the amount of technical work required for our ((necessarily lengthy) proofs, and the difficulty of partitioning this content petween the main body and the appendix. Our aim for the main body of the paper was to present and to explain and connect the various technical results which are very hard to state in a simplified fashion. We take your point and will add a content overview style paragraph/picture to the paper (or the appendix if space does not permit this).

1. What´s the point of figure 1a? Illustrating the degradation of the models as they are retrained on synthetic data? What is the difference betzeen the solid and the dashed curves?

The figure illustrates three key aspects: the degradation with respect to the number of generations, the effect of regularization on accuracy, and the scaling law curve concerning the amount of data. The dashed curves represent theoretical predictions, while the solid curves depict experimental results.

2. What is $T$ exactly?

$T$ represents the number of samples generated by the generator to train the subsequent model, while $T_0$ denotes the number of samples used to train the previous generators. The complete definition can be found in Equation (6) and please see Figure 3 for an illustration of all our notation. We will enhance the caption of Figure 1 - thank you for pointing out the possible unclarity.

3. I have a clarification question on the the theoretical setup: is it a semi-supervised setup, where you assume that you have access to $Y_0, X_1, X_2, ..., X_n$?

No. We consider the fully supervised setting. We use $X_i$ to denote the input data at iteration $i$, and we only have $\bar Y_i, X_i$. In the paper, we define the data and labels used to train the subsequent models in Equation (5) within the box, and we further elaborate on this in Figure 3.

4. On Theorem 3 (Equation 15), it seems that there is a remaining error $n \sigma_0 \rho$, even if the model is perfect at step 0. In other words, is it normal that if the model is perfect at step 0, then the test error still increase with the number of retraining?

Yes. Consider the groud truth data as generated from a perfect model, and our theory still apply. The test error of consequent models still increase since there are finite sample bias and label noises.

We thank you again for the appreciation of our work and are at your disposal if there are more questions. We hope our replies find your approval and would incite you to raise your score. Thank you.

We thank you for your thoughtful comments, and are delighted you appreciate the technical heavy lift that was required to prove our results and paint a picture of the regimes of "model collapse" in the setting of ridge regression. Thank you for pointing out that we should highlight what other technical arguments would fail (like martingale arguments) - we are adding this point to the next version!

1. The main weakness of the paper is that it seems only tangentially related to the generative model collapse problem ... Therefore, experiments using deep learning models supporting the theoretical thesis of this paper are very important for this work to appear applicable to the generative model collapse problem.

We addressed it in the general response.

2. The abstract is very vague and doesn’t describe what actually happens in the paper.

It is not mentioned in the abstract that the paper seems to focus primarily on Gaussian data, an important point. Moreover, “... quantitatively outline this phenomenon in a broad range of regimes.” Please say what regimes you are talking about otherwise this statement doesn’t really tell the reader almost any information. You mention “how model collapse affects both bias and variance terms” which is a very general statement but does this happen no matter what the model is? Or only with your specific setup? The evidence in the paper points to the latter and thus should be mention so that a reader does not think that this is a very general result.

Thank you for the suggestion! We have modified the abstract accordingly, with the differences marked in bold.

The era of proliferation of large language and image generation models begs the question of what happens if models are trained on the synthesized outputs of other models. The phenomenon of "model collapse" refers to the situation whereby as a model is trained recursively on data generated from previous generations of itself over time, its performance degrades until the model eventually becomes completely useless, i.e. the model collapses. In this work, we investigate this phenomenon within the context of high-dimensional regression with Gaussian data, considering both low- and high-dimensional asymptotics. We derive analytical formulas that quantitatively describe this phenomenon in both under-parameterized and over-parameterized regimes. We show how test error increases linearly in the number of model iterations in terms of all problem hyperparameters (covariance spectrum, regularization, label noise level, dataset size) and further isolate how model collapse affects both bias and variance terms in our setup. We show that even in the noise-free case, catastrophic (exponentially fast) model-collapse can happen in the over-parametrized regime. In the special case of polynomial decaying spectral and source conditions, we obtain modified scaling laws which exhibit new crossover phenomena from fast to slow rates. We also propose a simple strategy based on adaptive regularization to mitigate model collapse. Our theoretical results are validated with experiments.

Results are pretty much all asymptotic. This is kind of hinted at by the mentioning of “scaling laws” but needs to be more explicit in order to reflect the work of the paper.

We mentioned that "In this work, we study this phenomenon ..., under low- and high-dimensional asymptotics".

3. It is not clear what was mystified and now has been cleared up…

We have discussed how our theory compares with previous theories in the general response. Generally speaking, the concept of model collapse, as popularized by prior works, still lacks a theoretical framework to support empirically-backed findings. Our work addresses this gap by developing an analytic theory that clearly demonstrates how model collapse quantifiably emerges from training on synthetic data. This is reflected in the 'demystified' aspect of the title of our paper. We will incorporate this comparison into the paper.

4. Formatting

Thanks for the suggestion. We will change the font size.

5. Typos and mistakes:

Thanks a lot! We have corrected them.

We would like to thank you once again for reviewing our work and helping us improve its presentation, especially with regard to weakness No.2. Please let us know if you have any more questions. If there are no outstanding concerns, we would kindly ask you to consider raising your score which would substantially help in reaching a reviewer consensus. Thank you.

We thank the reviewer for the further engagement.

1. Regarding using the word "demystified"

We are happy to accept the proposed title to reduce the emphasis on "demystify."

2. Regarding how these results apply practically

I don't think that the authors mean to say that these test error formulations that they develop hold for multi-layer neural networks in regression settings as well, do they?

Let us clarify. Regarding complex neural network models

• Our theory (i.e linear increase in test error as a function of the number of iterations $n$) predicts their behavior in the linearized regimes (finite-width random features and NTK regimes, etc.). This is because such models are essentially kernel regression [1, 2]. Specifically for the RF regime, we empiriclally confirm this with additional results given in Table 1 below, which complement the results of Appendix C.2 of our manuscript (corresponding to classical RBF and polynomial kernels).
• For neural networks with finite widths (and fully trained), our theory doesn't direcly apply. Our speculation is that such a collapse continues to occur (consistently with empirical observations in the literature, albeit for LLMs based on transformers). We anticipate that the general trends uncovered by our asymptotic theory will hold true—for example, more parameters is expected to lead to greater model collapse, as shown in Theorem 3.1 and demonstrated in the following Table 2. More on this latter. Thus, model collapse in neural networks is even more severe than in linear regression.

However, it is quite possible that for general NNs (finite width, full-trained, etc.) amount of model collapse as a function of number of iterations $n$ might switch between linear increase (our current theory) to, quadratic, etc., depending on properties of the network like the the width, or other design choices.

To explore this hypothesis further, we conducted a series of experiments over the past 24 hours. Our aim was to extend the analysis beyond linear settings and Gaussian data by employing a two-layer neural network with ReLU activation on the MNIST dataset. The models were trained using stochastic gradient descent (SGD) with a batch size of 128 and a learning rate of 0.1. We employed a regression setting where labels were converted to one-hot vectors, and the model was trained using mean squared error for 200 epochs to convergence. For each data synthetic generation, Gaussian label noise with a standard deviation of 0.1 is added. The test error is consistently evaluated on the test set using the clean labels.

We considered two scenarios:

• (1) learning with random features (RF) models, where the first layer was fixed randomly, and only the second layer was trained, and
• (2) learning with a fully trainable neural network.

The results for RF models of width (i.e number of hidden dimensions) $k$ of 20,000 are presented in the table below (figures cannot be uploaded). For clarity, an additional column showing the test error gap between different generations (indicated by $n$) is included.

Table 1. Performance of very wide RF model on MNIST, with one-hidden layer NN (width $k=20,000$). Standard deviation is calculated using 10 random seeds.

We observe that, with the exception of the first two generations, the decay in MSE loss generally follows a linear trend, which is consistent with the predictions of our theory.

[8] Neel Nanda, et al. "Emergent Linear Representations in World Models of Self-Supervised Sequence Models." EMNLP 2023.

[9] Kiho Park, et al. "The Linear Representation Hypothesis and the Geometry of Large Language Models." ICML 2024.

[10] Yibo Jiang, et al. "On the Origins of Linear Representations in Large Language Models." ICML 2024.

[11] Yunzhen Feng, et al. "Beyond Model Collapse: Scaling Up with Synthesized Data Requires Reinforcement."