On the Interpolation Effect of Score Smoothing

1. A typo at line 57: $t$ should be inside $\sqrt{\cdot}$.

2. The motivation for the chosen local smoothing technique is not entirely intuitive. While they study an approximation in the $L^2$ sense, it remains unclear why this specific smoothing function is preferable over alternative smoothing methods. The paper acknowledges this limitation but could further clarify. See questions below.

3. The relationship between $\kappa$ and $t$ in section 3 is a bit confusing. See questions below.

Major questions:

1. The biggest concern is the gap between the problem that the authors attempt to study and the model proposed in this paper. The starting point is memorization vs. generalization in diffusion models. The authors conjecture the memorization is from the smoothed score function. It raised two natural questions: 1) how do you know the NN-learning score estimator has such smoothing effect in practice? 2) Even though the score estimator used in practice is smoothed, how is this property linked to the generalization, i.e., overcoming the memorization? These two questions are not well-studied in the paper.
2. The paper considers a(n) (essentially) one-dimensional model. In this case, the (empirical) score function has a closed-form. However, the authors do not study this score function while turning to its smoothed version. Then I would like to ask, even if we know all the properties of the smoothed score function, how can we have more knowledge about the true score function, even in this simple case?
3. The authors conduct numerical experiments to compare three score functions/estimators. However, it is not clear how to interpret the results. The distribution after adding noise must be the same as standard Gaussian is the stationary distribution in all three cases. Then, what is the new information by plotting the density and histograms?

Minor questions:

1. Why do you introduce $\hat{x}_t^{(n)}$, $\bar{s}_t^{(n)}$ and $\widehat{s}^{(n)}_{t, \tau}$, three versions of the score function? Also, if the target distribution is $p_0^{(n)}$, why to you need a smoothed version $\hat{p}_t^{(n)}$? Please clarify the relation between them and emphasize the necessity of introducing these auxiliary functions/distributions.
2. The target distribution studies seems to be relevant to Gaussian mixture models. There are some papers studying learing GMMs using diffusion models, e.g., https://arxiv.org/abs/2404.18869 and https://arxiv.org/abs/2404.18893. Any relation to the literature?

It is not clear how the analyses of one one-dimensional subspace can be extended to more practical data.

• For example, how can the analyses be connected back to address memorization/hallucination behaviors mentioned in the introduction?
• In line 475, I cannot agree the analysis is "distribution-agnostic," as it still relies on a highly simplified assumption of the practical data.

I think the paper could do a better job at motivating their results for a wider audience:

1. Equation (8) can and should be defined before Lemma 1 , in Lemma 1 you throw in the smoothed score defined in an implicit way and then you redefine it and label it it equation 8, it would be better to introduce equation 8 first and then use the notation defined in equation 8 to introduce lemma 1.
2. It seems plausible that a network would learn a smoothed version of the score (as discussed in limitations), but you never strongly motivate / discuss this, earlier discussion of this would motivate your choice of analysis better. Either way it would be helpful for you to motivate and introduce the smoothed score a bit earlier in the text.