The Inductive Bias of Minimum-Norm Shallow Diffusion Models That Perfectly Fit the Data

C: The nature of the work gives limited future work directions, as it is mainly deriving properties of specific closed-form solutions. I do not see how to generalize this work to e.g. minimum RKHS norm solutions, NTK, weight regularization etc., as the form of robustness considered is quite nonstandard.

R: We would like to clarify the following relation between the min-cost solution and the directions suggested by the reviewer:

(1) Weight regularization: The min-cost solution can be defined as the global minimizer of the loss with an infinitesimally small regularization on the weights (but not on the biases and skip connection). Therefore, our results already match those of training with such weight regularization (see `general comment’ on this, and the new results in Appendix D)

(2) RKHS norm, NTK: As proven in [A], the min-cost solution cannot be written as an RKHS norm (it is closer to an L1 norm). Thus, the min-cost solution cannot be an NTK solution. Therefore, it is impossible to generalize our results in this direction.

However, we see many other directions to generalize our work, such as understanding the effect of the scheduler, variance preserving probability flow (in contrast to the variance-exploding version we examine here), and the number of noise samples per training point.

[A] Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case

C: It is unclear how the simple observation of Prop. 1 contributes to the work, as well as the comment afterwards: ``training points have high probability''. This does not follow from Prop. 1 at all, as being a stable stationary point does not imply high probability.

R: We thank the reviewer for raising this important point. Proposition 1 demonstrates that training points are stable stationary points of the score flow, which represents the sampling process from one of the modes of the (multi-modal) distribution of $y$. This means we expect the score flow to converge to the training points, and it does not necessarily imply a high probability in cases where additional stationary points exist. However, in the specific case where the score function is differentiable and the training points are the only stationary points, it follows that the score flow will converge to the training points with probability 1 since the probability of reaching non-stable stationary points is zero for any non-measure zero initialization (from standard properties of gradient flow). The connection to the rest of the work lies in our observation that in probability flow the training samples are also stable stationary points. We have clarified this in the revised version (lines 206-207).

C: The experiments seem to introduce "diffusion sampling". Is this probability flow or something else? If it is, then this should be made clear. If not, diffusion sampling needs to be defined and probability flow should show up somewhere in the experiments.

R: Yes, in “diffusion sampling” we meant the discrete probability flow. We have fixed this in the revised version.

C: Other typos and suggestions (p.5; Eqs 17,19; Sec 4.2; Tweedie’s identity)

R: We thank the reviewer for the suggestions and fixes. We have revised the paper accordingly.

C: “what are the hyperbox defined by the data and its boundary?”

R: A hyperbox is an $n$-dimensional analog of a rectangular. For example, if we have 3 orthogonal training points $x_1$, $x_2$, and $x_3$ the vertices of the hyperbox are $x_1$, $x_2$, $x_3$, $x_1+x_2$, $x_1+x_3$, $x_2+x_3$, $x_1+x_2+x_3$.

C: “what's the definition of normalized score flow mentioned in the second paragraph in Section 4.2?”

R: The normalized score is the score function, normalized for visual clarity purposes (only in Figure 1). Specifically, we multiplied by the log of the norm of the score and divided by the norm of the score. We moved the definition to the main text in the revised version (lines 286-287)

C: “quantities like $\rho_0(\epsilon),T_0(\epsilon,\rho),T_1(\rho)$ in Theorem 2, and $\tau(y_T,\rho_T)$ in Theorem 3 lack interpretation. Similar for other quantities in the other theorems; 2)..”

R: To simplify the theorems we do not use the full analytical expressions, which can be found in the Appendix B2&B3.

C: “.the assumptions on ${u_i}_{i=1}^n$u_i, and the use of the Taylor's approximation in the small-noise level regime require more justification and discussion. ”

R: We added Figure 7 in Appendix E a 2D plot of the trajectory of the exact score function and the trajectory of the approximated score function. As can be seen, the trajectories are practically identical for a small noise level.

C: “the meaning of "converge to" is unclear. For example, in Theorem 3, by "converge to this vertex", does it mean that $y_0$ is exactly the vertex?”

R: In this paper, we analyze the solutions of the probability flow ODE and the score flow ODE. The probability flow ODE represents the reverse process in diffusion models. For this case, we study the solution over the time interval $[0,T]$, where the sampled point is at $y_0$​ at $t=0$. In contrast, for the score flow ODE, we analyze the solution for $t>0$, focusing on the behavior as $t \to \infty$. In the probability flow ODE the process arrives at the point exactly. Since this is a special case of convergence and to maintain consistent terminology, we used the term "converge to a point" in both cases.

C: “The three types of training data seem artificial and it is not clear how they are related to real-world data.”

R: The orthogonal datasets are commonly used in the literature as an approximation for random data with isotropic distribution in high-dimensional spaces e.g. [A,B,C] below.

[A] Andrew M. Saxe, James L. McClelland, Surya Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

[B] Etienne Boursier, Loucas Pillaud-Vivien, Nicolas Flammarion, Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

[C] Konstantin Donhauser, Mingqi Wu, Fanny Yang, How rotational invariance of common kernels prevents generalization in high dimensions.

C: "One concern I have about the scope of the work is that it operates in the "low noise" regime without explicitly defining what this entails"

R: We define the "low noise regime" as defined in [A]: a scenario in which clusters of noisy samples remain well-separated around each clean point. This setting includes non-negligible levels of noise. For example, in the BSD68 denoising benchmark, a widely used benchmark in denoising, a noise level of σ=0.1 is considered to fall within the low noise regime, as discussed in [A]. Furthermore, recent findings indicate that the "useful" aspects of diffusion dynamics predominantly occur below a critical noise threshold [B]. More specifically, the simulations in Section 5 include 150 denoisers, of which only the (last) 50 timesteps (those with the smallest noise level) the noise balls around each clean data point do not intersect, nevertheless, the simulation confirmed our theoretical findings. We have clarified this in the revised version (see footnote 1 lines 160-161).

[A] Chen Zeno, Greg Ongie, Yaniv Blumenfeld, Nir Weinberger, and Daniel Soudry. How do minimum-norm shallow denoisers look in function space?

[B] Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models.

C: "I would also encourage the authors to further explore the effect of early stopping."

R: Please note that we do not use early stopping in our simulation. By “early stopping” we refer to the inherent effect of the scheduler on the probability flow sampling process.

C: “Additionally, could the authors comment on what they might expect if the minimal norm assumption were lifted?”

R: We thank the reviewer for this important question. We added in Appendix D additional simulations where we train the neural denoisers using a standard training procedure with the Adam optimizer, with or without weight decay regularization. Specifically, for weight decay, we tried $\lambda=0.25, 0.5, 1$. All values resulted in similar results, therefore we included in Figure 5 only the case of $\lambda=0.25$. As can be seen, in the case of WD=0 we converge only to the training points or boundary points of the hyperbox, whereas for WD>0 we converge to virtual points as well, which aligns with the results achieved with the Augmented Lagrangian method (which we used to obtain the min-norm denoiser).

C: “While the results present in the paper have an important connection with memorization/duplication in diffusion models, the presented work does not take the number of samples into account.”

R: We thank the reviewer for pointing that out. Based on the reviewer's suggestion, we added in Appendix C simulations to analyze the effect of the number of training samples on the diffusion sampling process. Specifically, we repeat the experiment from Section 5 while changing the number of training points $N$. Figure 4 shows that, as we increase the number of training points, the probability flow converges less to training points, and more to boundaries and virtual points, i.e., the generalization increases. Regarding the effect of oversampling duplications, previous papers observed that diffusion models tend to overfit more to duplicate training points than to other training points. However, note that we study the regime in which the model anyway perfectly fits all the training points. In practice, if the model does not perfectly fit all training points, duplicating some training points would cause the network to fit them better, at the expense of the other training points. This suggests our analysis still holds, but only for the well-fitted training points and their associated virtual points. Mirroring the previous experiments' results, this decreases the generalization and will cause more convergence to the duplicated training points.

C: "Line 430: What is meant by The decrease in percentages is due to small deviations in the ReLU boundaries of the trained denoiser compared to the theoretical optimal denoiser."

R: In practical training, the ReLU boundaries have some small deviations from the closed-form solution. If there are small deviations in the ReLU boundary, the neural network's output h(x_i+x_j) will no longer equal x_i+x_j​, which is a necessary condition for a fixed point. The impact of small deviations in the ReLU boundary is analyzed in Theorem 4.

C: "Line 442: Why is 0.2 set as the l_inf threshold? Could the authors show how continuously changing this value affects the distribution of data points"

R: We show the effect of changing the $L_{inf}$ threshold in Appendix E, as well as for $L_2$ threshold. Qualitatively, this does not drastically affect the results.

C: "Line 470: specifically in our example for the first 100 denoisers, the noise level is not small compared to the norm would the authors be able to elaborate this part for clarity? Why is the noise level small for only the first 100 denoisers? Why are they not expected to have stable virtual points?"

R: In this work, we analyze the variance-exploding version of the probability flow. There, theoretically, the reverse process begins with Gaussian noise of infinite variance. In practical sampling (as shown in Section 5), we initiate the sampling process with a large, but finite, variance for the Gaussian noise. In the analytical derivation (Section 4), our results hold starting from the point in the reverse process where the noise balls around each clean data point no longer intersect, which is what we refer to as the ``low-noise regime’’. The simulations in Section 5 include 150 denoisers, of which only the (last) 50 timesteps (those with the smallest noise level) the noise-balls around each clean data point do not intersect. Therefore, stable virtual points are not expected to form for the initial 100 denoisers. We have clarified this in the revised version, thanks (lines 466-473).