We sincerely appreciate the reviewer’s positive feedback and are glad that they find this a fascinating contribution. The interplay between memorization and generalization is indeed intriguing.

Low-Noise Regime Details

Theoretically, the boundary of the low-noise regime is determined by the diffusion time scheduler, the number of timesteps, $T$, the training data, and the noise realizations. In the experiments, we approximate this boundary by considering a ball radius equal to 4 standard deviations. Specifically, we first calculate the variance schedule of the $T=100$ diffusion timesteps. Then, we check the minimal distance between training points. We consider the low-noise regime boundary to be an eighth of this distance. Once we have this boundary, we extend the variance schedule to include at least $50$ variance values below the boundary.
Since we are discretizing an ODE, as long as we are using small enough timesteps, we would get the same results. We will clarify this in the revised manuscript. In practical models, we are guaranteed to have some denoisers in the low-noise regime, since the final sampled point is obtained when $t \approx 0$, where the noise level is inherently small.

Additionally, the low-noise regime is practically important for diffusion sampling, as the "useful" part of the diffusion dynamics occurs primarily when the noise level drops below a certain critical threshold [R1].

Implications for Practical Image Diffusion Models

In practice, natural images lie on a low-dimensional linear subspace due to their approximate low-rank structure (see Zeno et al. (2023), Appendix D3). As a result, the denoiser first contracts the data towards this subspace (see Theorem 2 in Zeno et al. (2023)). If the subspace has sufficiently high dimension, the data points become approximately orthogonal (following the $\exp(D) \gg N > D \gg 1$ argument in our response to Reviewer SMjF under “Orthogonal Dataset Choice”). Consequently, the geometric structure in high-dimensional image spaces might resemble the hyperbox studied in our work, though restricted to a subspace, and with slightly acute angles rather than strictly orthogonal ones.

Reliance on Zeno et al. (NeurIPS 2023)

We acknowledge the reliance on Zeno et al. (NeurIPS 2023), but we believe that advancing the understanding of diffusion models remains valuable, even within these constraints. We believe that exploring alternative data arrangements is an interesting and challenging direction for future work.

Fig. 3 Points Order

Thanks for this valuable suggestion, we will swap the order in the revised paper.

Convergence Inside the Hyper-Box

We do not converge inside the hyperbox because, as shown in Fig. 8 (Appendix F), the score function trajectory, both inside and outside the hyperbox, always points toward its boundary. As a result, the probability flow ODE leads to convergence at the hyperbox boundary rather than within its interior.

[R1] Gabriel Raya & Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models. NeurIPS, 2023.

We greatly appreciate the reviewer’s kind words. We are glad to hear that the paper is presented in an accessible and engaging manner and that it was enjoyable to read.

Application to Higher Dimensional Data

Indeed, we make simplifications to allow tractability. Please note, however, that while $N>D$ makes it impossible for the dataset to be exactly orthogonal, in the realistic regime $\exp(D)\gg N>D\gg 1$ most pairs are nearly orthogonal for standard Gaussian data. We believe this is a main reason why the orthogonality assumption is reasonable and common in high-dimensional theoretical analysis.
Please see more details in our response to Reviewer SMjF under “Orthogonal Dataset Choice”.

Manifolds With Curvature

We appreciate this insightful suggestion. Our theoretical analysis is based on Euclidean geometry and does not easily extend to nonlinear manifolds with curvature. Investigating how score and probability flows behave in such spaces is indeed an important direction for future research. Additionally, identifying meaningful experimental quantities analogous to those we defined (e.g., virtual points) in curved manifolds poses an interesting challenge.

Dropout Effect

The impact of dropout on the generation of virtual points depends on several factors, including whether it is applied to the input, hidden layers, or both, as well as the dropout rate. Could the reviewer kindly elaborate on the specific dropout setting they have in mind?

Results for Additional Datasets

We thank the reviewer for the suggestion. We will add additional visualizations to the appendix and highlight these distinctions in the revised version of the paper.

Connection to Hopfield Models

We appreciate this insightful suggestion. In the revised version of the paper, we will incorporate the Memorization, Spurious, and Generalization metrics into our experimental results.

Essential References

We thank the reviewer for these references; we will discuss these related works in the revised manuscript.

We sincerely appreciate the reviewer’s positive feedback.

Orthogonal Dataset Choice

First, when $N>D$, it is indeed impossible for the dataset to be exactly orthogonal. However, for standard i.i.d. Gaussian data $x_n$, it is easy to show (e.g., using the analysis in Section 3.2.3 of [R1] and the union bound) that the largest cosine of the angle between two different datapoints is, with high probability, $\max_{n\neq m} \frac{x_n \cdot x_m}{\|x_n\| \|x_m\|} \sim \sqrt{\frac{\ln N}{d}}$. Therefore, in the realistic regime $\exp(D) \gg N>D \gg 1$, most pairs are nearly orthogonal. We believe this is a main reason why the orthogonality assumption is reasonable and common (e.g., [R2]) in high-dimensional theoretical analysis.

Second, to make sure we are on the same page, please note that our analysis goes beyond the orthogonal case and covers all the analytically solvable cases found in Zeno et al. 2023, though some of this analysis was relegated to the appendix. For example, in App. D (line 630), we extend our analysis beyond the strictly orthogonal case by considering the scenario where the convex hull of the training points forms an obtuse-angle simplex. We showed there that similar behavior emerges even when strict orthogonality does not hold, supporting the broader relevance of our findings. In the main paper, we indeed focused on orthogonal datasets: this was done to simplify the presentation, and since the solution is qualitatively similar to that in the obtuse angle case.

Interpolation Meaning

Yes, by "interpolates the training data," we mean that the denoiser maps each noisy sample to its corresponding clean training point, which leads to a zero empirical loss. Solutions with zero empirical loss are commonly called interpolators in the ML theoretical literature (e.g., [R3]). As an approximation to this, we further assume that the denoiser maps the entire open ball centered around each clean data point to that clean point (this was also done and explained in Zeno et al.).

Virtual Points & Generalisation

Yes, we view these virtual points as desirable, since they go beyond the empirical data distribution, and create new combinations not seen before in the training data. It is perhaps more accurate to view this on some scale: low-order combinations (e.g., which can be easily observed as “copy-paste”) may seem closer to memorization (“low creativity”), while high-order combinations show a higher degree of generalization (“high creativity”). We will discuss this interesting point in the paper.

Generalisation in Orthogonal Datasets

In the case of an orthogonal dataset, we interpret the hyperbox boundary as an implicit data manifold, even though we do not assume an explicit sampling model that generates the training data (e.g., a distribution supported on the manifold). In this context, we define generalization as the ability to sample points from the hyperbox boundary.

Noise Distribution Modeling

In practice, diffusion models are trained for a finite number of iterations using Gaussian noise. Consequently, during training, we effectively observe (different) noisy samples that all lie within an open ball around each clean data point (see additional details in sec. 3 in Zeno et al. 2023). In the multivariate case, finding an exact min-cost solution for finitely many noise realizations is generally intractable. So, we assume that the denoiser maps each open ball to its corresponding clean data point.

[R1] Roman Vershynin. High-Dimensional Probability. 2019.

[R2] Andrew M. Saxe et al. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv:1312.6120, 2013.

[R3] Siyuan Ma et al. The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning. ICML, 2018.

We thank the reviewer for their thoughtful feedback.

Connection to Semantic Sums

We are sorry for the lack of clarity. We did not aim to claim that the virtual points in orthogonal data must be equivalent to combinations of semantic components of images as observed in Stable Diffusion. This is only maintained as a motivation for virtual points. We agree that the current phrasing in the abstract may be confusing, and will change this in the revised paper.

Relaxation of Obtuse Angles

The same arguments used in the proof of Theorem 3 in Zeno et al. (2023) can be applied to prove equation (17). The requirement for strictly obtuse angles (i.e., $x_i^Tx_j < 0$ instead of $x_i^Tx_j \leq 0$) in Zeno et al. (2023) is only made specifically to ensure the uniqueness of the solution. The proof of that theorem, therefore, applies to the orthogonal case as well, just without uniqueness. In other words, the function in (17) is a minimizer of (8), but not necessarily the unique minimizer.

Specifically, the proof of Theorem 3 in Zeno et al. (2023) uses the fact that any unit active along two data points making an obtuse angle can be “split” into two units aligned with each data point with strictly lower representation cost. When the rays are orthogonal, the two “split” units may have the same representation cost as the original unit. So while the interpolant given by equation (17) is still a minimizer, it is no longer necessarily unique in the case of orthogonal data.

We will clarify this in the paper.

Constraining Denoisers to Exactly Fit the Training Data Outside the Low-Noise Regime

Indeed, outside of the low-noise regime, the denoisers would not exactly fit the training data, and the min-norm denoiser in this case is not defined (as we can not get zero loss). We use the same training regime for all denoisers for consistency. To complement that, we conduct additional simulations in App. E, where we train the neural denoisers using a standard training procedure using the Adam optimizer with weight decay, obtaining similar results. We hypothesize that this is because high-noise denoisers primarily influence the initial condition, before the critical noise threshold is crossed, and we start the “useful” final sampling phase [R1].

Strength of Assumptions

Indeed, we make simplifications to allow tractability. Please note, however, that while $N>D$ makes it impossible for the dataset to be exactly orthogonal, in the realistic regime $\exp(D) \gg N>D\gg 1$ most pairs are nearly orthogonal for standard Gaussian data. We believe this is a main reason why the orthogonality assumption is reasonable and common in high-dimensional theoretical analysis.
Please see more details in our response to Reviewer SMjF under “Orthogonal Dataset Choice”.

The Constrained Optimization Procedure

App. E holds additional simulations where we train the neural denoisers using a standard training procedure with the Adam optimizer, with or without weight decay regularization (which promotes min-norm results, but not forces it). Specifically, for weight decay, we tried $\lambda=0.25,0.5,1$. All values led to similar results, therefore we included in Fig. 6 only the case of $\lambda=0.25$. As can be seen, in the case of WD=0 we converge only to the training points or boundary points of the hyperbox, whereas for WD>0 we converge to virtual points as well, which aligns with the results achieved with the Augmented Lagrangian method.

Diffusion Time Scheduler

Thanks for this suggestion, we will change the terminology in the revised paper.

Min-cost Denoiser Independent of $t$

The time dependence of the min-cost denoiser is through $\rho$. Specifically, in eq.11 (probability flow ODE), the RHS includes a derivative w.r.t $t$ of $\sigma_t^2$ times the score function, so by applying time re-scaling arguments, we get a time-dependent $\rho$, and, therefore, denoiser. In eq. 13, however, the RHS does not include a derivative with respect to $t$, so $\rho$ and the denoiser are time-independent.

Significance of the 0.2 Threshold

We show the effect of changing the thresholds values in App. F. Qualitatively, this does not drastically affect the results.

Samples Beyond Training/Virtual/Boundary Points

No, all points converged to one of the 3 categories, which matches our theoretical analysis.

Extending Fig. 3 to $N>d$

We thank the reviewer for this suggestion. In strictly orthogonal data, it is impossible to have $N>d$. If the reviewer has a different dataset in mind, we will be happy to train all our denoisers (150) on the relevant $N$ and $d$ on this new dataset, and include our findings in the revised paper. Note also that if we change the dataset, it can become challenging to define and find the virtual points.

[R1] Gabriel Raya & Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models. NeurIPS, 2023.