Diffusion Models Learn Low-Dimensional Distributions via Subspace Clustering

• I believe the authors should highlight the paper's contribution. Whether the modeling is justified or not, I believe the paper should contain a meaningful takeaway message. For example, some numerical relationship between the denoiser's jacobian's rank and the number of samples required.
• The experiments conducted do not provide sufficient convincing evidence that the chosen modeling is fitting. Moreover, it is unclear what supports the application of the conclusion following Theorem 4 to real data.
• I am uncertain why modeling the DAE as a mixture of zero-meaned Gaussian is justified. Assuming that the data is a union of linear subspaces, using low-rank Gaussians is reasonable yet the subspaces do not necessarily coincide at the origin. Could the authors please shed more light on this choice?
• The use of the principle components of a Denoiser's jacobian for semantic exploration has been explored in previous work, namely [1]. Also, the connection between the proposed MoLRG and the semantic correspondence of the DAE's jacobian's principle component was unclear.

[1] Hila Manor & Tomer Michaeli (2024). On the Posterior Distribution in Denoising: Application to Uncertainty Quantification. In ICLR 2024.

1. The main weakness of the paper is that the main limitation of the analysis is not acknowledged clearly enough. This limitation is due to the main assumption of the work: image data lies on a fixed union of low dimensional sub-spaces. Although better than a full-rank Gaussian model, this is still a very crude approximation. As cited by the authors, image data lies on a union of non-linear manifolds, which cannot necessarily be accurately approximated by a union of linear manifolds (sub-spaces). The nice linear relationships between N and d will not hold as soon as you have non-linear manifolds. For example, for k =1, you would need N=2 to get a perfect estimate of a one dimensional subspace with a linearity assumption. But as soon as you have a non-linear manifold, depending on the degree of non-linearity you would need larger N. I suggest the presentation should be modified to reflect the limitation of the analysis. Otherwise, the way the paper is written now, sets up the reader for disappointment, as the results do not extend to the real image data as claimed by the authors. Nevertheless, I think the results are valuable regardless of the simplistic assumption. They just need to be upfront about the assumptions.
2. Modeling image distribution with a union of sub-spaces is an old idea in signal processing that goes back to wavelet thresholding and later compressive sensing literature. The paper would benefit from citing major papers where these ideas originated. Importantly, the optimal solutions under this kind of assumption has been a very active topic in that area. It would be interesting to see how these results connect to that literature.
3. Another assumption made is to approximate the weights with a hard-max operation. This is of course a very hard assumption that results in a lot of error for high noise levels (when noisy image is far from the union of subspaces). Importantly, this assumption simplifies the posterior mean too much, which is counter productive when the goal is to explain diffusion models (where you have large levels of noise).
4. There seems to be a confusion in the experimental results presented in Figure 5. As a reader, I expect the experimental result to support the theory. However, the generalization result rely on a generalization score that is defined in the appendix. The results shows that the generalization as defined in eq 48 is related to N_k/d_k in a sensible way. However, it does not support nor refutes the results presented in the 4 theories (main results). So there is a divide between the theoretical results and the empirical results which is supposed to support the theory.

Weaknesses:

1. It is not a new idea to study behaviors of diffusion models, or generative models in general, along subspaces spanned by leading eigenvectors of the Jacobian. I cannot produce a complete bibliography in this area, but the following papers may be relevant:

• Subspace Diffusion Generative Models, Jing et al., ECCV 2022 , which developed a new diffusion model that restricts the flow vector field to a linear subspace whose dimension shrinks as t->0
• The Geometry of Deep Generative Image Models and its Applications, Wang & Ponce, ICLR 2021, whose experiments focused on GANs rather than diffusion models, but revealed that generative models identifies top eigenspaces of the Jacobian, which capture important perceptually relevant changes.

2. The study in this paper assumes the distribution is a sum of finitely many Gaussians centered at 0 with added noises (and even assumed later that they have the same weights). This is an oversimplification as under the current assumption, the task becomes an optimization of finitely matrices. The real-life diffusion is much more complicated as it tries to identify, instead of linear subspaces, submanifolds or equilvalently a tangent linear subspace at each point, which is an infinite dimensional task. How would the results change for data lying on low dimensional non-linear manifold? I guess the complexity of the task would depend on how fast the tangent space change among nearby points, which is quantified as curvatures of the manifold. It would be interesting to see discussions addressing this aspect.
3. This contrast is clearly demonstrated by the current paper's own experiments: under the simplified assumption, only a handful of samples (equal to the rank of the matrix) are needed for decent inference quality while for realistic data thousands of samples are needed ( Figure 5a vs Figure 5b ). While it is true the ranking of difficulty among different datasets coincide with that of intrinsic dimension, it should be recognized that a great amount of new details exist in data with higher instrinsic dimensions, and model quality a priori cannot be naively summarized by transferring the analysis on low-rank Gaussians in Theorems 1-4. I think it would be interesting to have more analysis on whether local intrinsic dimension are the only features that need to be handled in real work datasets, in particular, whether the shape and smoothness of the distribution also play a role.