We thank the reviewer for their careful review of our submission, and for recognizing our work’s “well-motivated theoretical foundation” and “fascinating insights”. We respond point-by-point below, and look forward to discussing further.

My primary concern is that the theoretical concept of the crossover point is only shown for particular synthetic cases. While the paper promises a generalization to more complex datasets, empirical proof is missing. While I understand that training diffusion models can be costly, I would have expected to see at least some experiments on small scales, such as CIFAR-10.

In our view, the memorization-generalization interplay in diffusion models is a holistic phenomenon with numerous interrelated causes (as evidenced by the diversity of explanations proposed in the literature [1-4]), and to develop a mathematical understanding of it, it is necessary to study it in a setting in which these different causes can be disentangled, precisely varied, and their effects examined. In this sense, our work aims to build a theoretical foundation for studying memorization in diffusion models. Due to this scope, we do not attempt to rigorously mathematically analyze real data and real networks in our work. However, we note (here and in the paper) that experimental results in our simplified laboratory setting reproduce phenomena observed in previous empirical work studying memorization and generalization in real diffusion models trained on real data (i.e., larger than CIFAR10 scale), see e.g. [5].

The paper writing is based heavily on mathematical symbols. While Tables 1 and 2 provide a good lookup, it is sometimes hard to follow without frequently looking up symbols from previous sections.

Thank you for the feedback; while our work is intended to be theoretical in nature, and therefore involves significant amounts of mathematics, we agree that the notation indeed needs improvement, and we commit to streamlining it in the revision. With the additional content page, we will also add descriptions of the relevant variables before using them, e.g., “the number of components $K$”.

If I understood the method correctly, it seems like the method requires reference models / multiple training runs with different model capacities to identify the phase transition in practice. Assuming one wants to train a larger diffusion model without a valid reference, one cannot a priori define a sufficient number of parameters to avoid memorization.

Regarding the 3rd point stated under weaknesses, is it correct that the model always requires multiple training runs with different capacities to identify and predict the crossover point? Or is it possible to predict the number of parameters for, e.g., a model trained on ImageNet?

Indeed, in our methodology, we need to build a predictive model by learning the loss weighting $\tilde{\lambda}$. However, as we show in the paper, we can use the learned loss weighting to extrapolate to different (held-out) training configurations, and predict the location of the phase transition from generalization to memorization with high accuracy. Notice that this is similar to the current literature on scaling laws, where a simple law relating the size of the model to some property (such as the performance on some benchmark) is fit using a number of trained models, and then used to predict the given property for different model configurations without training those models [6]. In this context we can reframe a contribution of this paper as: within our laboratory, we derive a hypothesized scaling law form for memorization and demonstrate that it holds to high accuracy in practice. We humbly point out that scaling law work conventionally requires this curve-fitting and cannot give a principled hypothesis as to the form of the scaling law. As a result, we do not view this limitation as a major weakness, since in fact it constitutes an improvement (a prediction about the form of the scaling law) compared to numerous works in this domain.

Please let us know if you have other questions or hang-ups, and we will be happy to resolve them. We look forward to a productive discussion during the rebuttal period.

References

[1] Kadkhodaie, Z., Guth, F., Simoncelli, E. P., & Mallat, S. (2023). Generalization in diffusion models arises from geometry-adaptive harmonic representation. In arXiv [cs.CV]. arXiv.

[2] Kamb, M., & Ganguli, S. (2024). An analytic theory of creativity in convolutional diffusion models. In arXiv [cs.LG]. arXiv.

[3] Finn, E., Keller, T. A., Theodosis, M., & Ba, D. E. (2025). Origins of creativity in attention-based diffusion models. In arXiv [cs.LG]. arXiv.

[4] Niedoba, M., Zwartsenberg, B., Murphy, K., & Wood, F. (2024). Towards a mechanistic explanation of diffusion model generalization. In arXiv [cs.LG]. arXiv.

[5] Zhang, H., Zhou, J., Lu, Y., Guo, M., Shen, L., & Qu, Q. (2023). The Emergence of Reproducibility and Consistency in Diffusion Models. In arXiv [cs.LG]. arXiv.

[6] Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D., 2020. Scaling laws for neural language models. In arXiv [cs.LG]. arXiv.

We thank the reviewer for their thorough review of our submission. We respond point-by-point below.

Given the clarifications below around the soundness of our methodology, we hope the reviewer will consider raising their score. We look forward to discussing any remaining issues further.

this paper claims a "phase transition" without really giving a good definition of what that is. It is sort of obvious that if you increase the memorization capacity of your model that your model will memorize more and generalize less (this is observed ubiquitously in machine learning) so merely providing plots and claiming that it is happening doesn't seem very compelling.

We respectfully yet strongly push back on the notion that we are “merely providing plots and claiming that [the phase transition] is happening”. The existence of this phase transition and even its universality across different models and hyperparameters has been very well established by prior empirical work [1–2]. A main contribution in our work is to isolate the factors underlying this phase transition and reproduce it in a synthetic setting amenable to theoretical analysis, and to empirically link the precise point at which it occurs in trained models to easily-computable properties of the loss landscape. This sheds light into the mechanisms underlying the phase transition in a novel way with regards to previous empirical studies, and can act as a foundation for further finer-grained studies in the future.

In general, we can clarify the writing to more clearly introduce the concept of a phase transition. This term is prevalent in physics/statistical mechanics and their connection to machine learning [5–6], and phase transitions between memorization and generalization have been empirically observed in prior works [1--2], which we have discussed in the introduction and related work sections of the submission. Quoting from [1], a foundational work in this area:

We show empirically that diffusion models memorize samples when trained on small sets, but transition to a strong form of generalization as the training set size increases… [t]he amount of data needed to reach this phase transition is very small… and depends on the image size and complexity relative to the neural network capacity.

--- [1], p. 9

We respectfully note that the reviewer's objection (“it is sort of obvious…”) could be identically levied at any of these foundational prior works.

The fact that this phase transition has caused surprise and research interest in the context of diffusion models (in contrast to its "[ubiquity] in machine learning") may be attributable to the fact that certain diffusion models seem to memorize more than equivalent generative models [3], and that the relationship between the trained model (a denoiser for different times $t$) and the algorithm for generating samples (a multi-step iterative refinement of a noisy image with the trained model) obfuscates the link between "overfitting" behavior of the denoisers on a given timestep's loss and the ultimate memorization behavior of generated samples. On a more abstract level, note that in the context of diffusion model training (i.e., Gaussian denoising), the unique optimizer of the training loss is the model that memorizes the training data, as we describe in Section 2 of the submission. This should be contrasted with the situation in other machine learning settings, e.g. classification, where there are typically an infinite number of models that achieve minimum training loss, and the key question is to understand which model is learned, given one's specific training algorithm [4].

Can you explain why optimizing the weights \lambda_t actually isn't "cheating"? I'm not claiming that it is, but it does feel fishy when one of the main claims of this paper is that the theory "predicts" a transition point and yet this prediction is made with some sort of explicit training.

In short: Optimizing the weights $\tilde{\lambda}$ is not “cheating” because it is a standard and accepted scientific methodology. It corresponds to hypothesizing that an empirical phenomenon follows a certain mathematical law, then fitting the free parameters of the law based on empirical data, and verifying whether the resulting law correctly predicts the phenomenon on held-out data. This is the same methodology that underlies the modern practice of scaling laws for training large models [7].

Below, we provide a discussion with full details in order to completely clarify the role of $\tilde{\lambda}$ in our methodology.

In our submission, we hypothesize that the memorization-generalization transition occurs when the weighted relative loss of two surrogate models (a generalizing model and a memorizing model) switches from positive to negative (the "crossover"), for some weighting. In this way, our experiments effectively fit a predictive model $f(N, d, K; \tilde{\lambda})$ (defined in eqn. 19) with parameters $\tilde{\lambda}$ for the number of model parameters $M_{\star}(N, d, K)$ at which memorization becomes prevalent (lines 242-251), and validates the accuracy of this predictive model on a set of held-out simulation data (lines 284–288).

We can also demonstrate that the effective predictive model implied by this parameterization is a simple linear function of the number of samples $N$, so the reviewer should not be worried about any hidden capacity to overfit. The two surrogate models’ loss approximations that we use in our predictive model are justified by Theorem 3.1, namely

As we approach the end of the discussion period, we wanted to check back in with you and inquire as to whether you have any remaining questions, comments, or concerns. If so, we are happy to discuss them; if not, we would humbly request a re-evaluation of your score.

We thank the reviewer for their careful review of our submission, and for recognizing our work’s “sound and rigorous theoretical analysis”. We respond point-by-point below, and look forward to discussing further.

The simplifications required for theoretical tractability (i.e., the GMM assumption) create a substantial gap between the analyzed model and those used in practice.

This is an unavoidable artifact of developing rigorous theory for a system as complex as diffusion models. Moreover, in spite of the apparent simplicity of this data model and its associated denoisers, even basic theoretical questions about, for example, the number of parameters at which models learned from the class of GMM denoisers tend to memorize, and the relationship between these trained models and the sampling-time behavior of the diffusion sampling algorithm, remain completely open.

More precisely, in the theoretical literature on diffusion models, the Gaussian mixture model (GMM) we study actually sits at the ‘frontier’: almost any algorithmic research question one can formulate around diffusion models remains open for GMMs. Specifically with regards to the behavior of trained models, [1] proves that the minimizers of the training loss over a properly-parameterized class of GMMs generalize (in our notation, analogous to the number of components $M$ being equal to the number of classes $K$), [2] proves the same for models learned by gradient descent for two-point mixtures of Gaussians, and [3–4] establish guarantees for learning properly-parameterized GMMs with non-gradient-descent algorithms. To the best of our knowledge, no prior work has studied the setting where the number of components $M$ may be significantly larger than the number of classes $K$, which is the essential setting to understand memorization in GMM diffusion models as they require $M \gg K$ parameters to memorize with low loss. In addition, because existing theoretical works apply only to the ‘properly parameterized’ setting where $M = K$, they are able to guarantee that trained models generalize at sampling time by applying existing results on diffusion model sampling under accurate score estimates (e.g., [5–6]). In our setting where $M \gg K$, it is generally intractable to exactly characterize learned models, which means that characterizing the sampling-time behavior of learned models (i.e., memorizing or generalizing) is a completely open problem – even the Gaussian case (i.e., $K=1$) is highly nontrivial [7]. Our work paves the way towards a rigorous theoretical understanding of this non-trivial question via the hybrid theoretical-empirical approach we employ, which correctly predicts the location of the phase transition between memorization and generalization in terms of simple properties of the loss landscape.

The novelty of the core insight feels somewhat limited. The finding that increased model capacity leads to memorization is well-established. This work's contribution is to formalize this intuition and provide a predictive formula within a specific, very simplified setting, rather than offering a fundamentally new conceptual understanding of the phenomenon itself.

We respectfully disagree with your characterization of this work’s contributions and novelty, and would phrase it as follows. As a theoretical work, our work’s main contributions are proposing the GMM “laboratory” as an environment for studying emergent phenomena in diffusion models, and demonstrating its utility in rigorously characterizing the phase transition between generalization and memorization. Although this is a “simplified setting”, we note above that the theoretical study of diffusion models in the GMM setting is completely open as far as memorization behavior at intermediate model sizes is concerned. In this sense, our work aims to build a theoretical foundation for studying complex phenomena in diffusion models, rather than simply adding another explanation in a general setting, and we demonstrate the viability of this in characterizing the phase transition.

To strengthen the claim that your framework captures a more general principle, have you considered an experiment with a simple neural network (e.g., an MLP) as the denoiser? While your analytical predictions for the crossover point would not hold, demonstrating that a standard network trained with gradient descent exhibits the same qualitative phase transition as a function of the number of neurons would partially reinforce the paper's relevance.

We thank the reviewer for this suggestion. Firstly, we would highlight that the GMM denoisers we train in our work (eqn.s (9) and (12)) are in fact primitive MLPs, as inspection of these formulae reveals, with a softmax nonlinearity and a specific weight-tying structure. With regards to more practical architectures, we note that this experiment has been performed in prior work in the literature, and the reviewer’s suspicions confirmed. Specifically, in [1, Figure 4], the authors demonstrate under a slightly different metric for memorization (which measures similarity using a fixed neural feature descriptor model instead of comparing $\ell^2$ distances) that U-net models trained on GMM data undergo a rapid phase transition as the number of training data is varied. The existence of these results led us to focus on theoretically characterizing the phase transition, as in the submission.

Please let us know if you have other questions or hang-ups, and we will be happy to resolve them. We look forward to a productive discussion during the rebuttal period.

References

[1] Wang, P., Zhang, H., Zhang, Z., Chen, S., Ma, Y., & Qu, Q. (2024). Diffusion Models Learn Low-Dimensional Distributions via Subspace Clustering. arXiv preprint arXiv:2409.02426.

[2] Shah, K., Chen, S., & Klivans, A. (2023). Learning Mixtures of Gaussians Using the DDPM Objective. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023).

[3] Gatmiry, K., Kelner, J., & Lee, H. (2024). Learning Mixtures of Gaussians Using Diffusion Models. arXiv preprint arXiv:2404.18869.

[4] Chen, S., Kontonis, V., & Shah, K. (2024). Learning general Gaussian mixtures with efficient score matching. arXiv preprint arXiv:2404.18893.

[5] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2023). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations (ICLR 2023).

[6] Benton, J., De Bortoli, V., Doucet, A., & Deligiannidis, G. (2024). Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization. In The Twelfth International Conference on Learning Representations (ICLR 2024).

[7] Hurault, S., Terris, M., Moreau, T., & Peyré, G. (2025). From score matching to diffusion: A fine-grained error analysis in the Gaussian setting. arXiv preprint arXiv:2503.11615.

As we approach the end of the discussion period, we wanted to check back in with you and inquire as to whether you have any remaining questions, comments, or concerns. If so, we are happy to discuss them.

Thanks for your answers. I acknowledge and appreciate that theory necessitates simplifying assumptions. Yet, my primary concern remains that the theoretical framework is simplified to an extent that it represents a significant departure from diffusion models. Consequently, I am not fully convinced that the derived insights substantially advance our understanding of the behavior of more complex and practically interesting systems.

The paper is technically sound. However, given the concerns about the impact and applicability of its findings, I will retain my borderline acceptance recommendation.