A Geometric Perspective on Diffusion Models

• One major concern is that the authors focus only on the VE case (although they mention that a similar analysis applies to a variance preserving SDE). While interesting, such limitation reduces generality of the claims. Some of the numerical observations might simply be artifacts of the considered SDE class, and do not provide information about diffusion models in general. Moreover, I find the different sections to be somehow disconnected; in certain cases some results are presented as original, while to the best of my understanding, they are restating known literature results. Finally, the connection to a “geometrical perspective” is very weak.

• Recall Observation 1: “The sampling trajectory is almost straight while the denoising trajectory is bent”. As explicitly mentioned in Proposition 2, the norm of the corrupted data will be much greater than the clean data. On the other hand, the denoising trajectory will have (by construction) a roughly constant norm at each step. Then observation 1 is almost trivial. Is it still valid for other SDEs? If not, what information is the proposition offer to the reader? Similar considerations apply for Observation 2. Also, I would advise the authors to rescale Fig 2b (maybe logarithmic y axes), since in the “region of interest”(small diffusion steps) it is very difficult to compare top and bottom subplots.

• Observation 3 is equivalent to the following rephrasing: “the output of a denoiser has better FID score than its input”. This is trivially true, and once again provides very limited insight into the geometry of diffusions. One interesting technique is related to the computation of the FID score using the JUMP scheme, but it seems that the technique provides limited benefit for values of FID which are competitive.

• Section 4 discusses how second order schemes use in their numerical implementation approximations of the second order time derivative of the sampling process. This is exactly how second order numerical schemes are derived. Then, what is the take home message of this Section?

• Proposition 4 is a known result from [Karras 2022], in particular Eq. (57) in the Appendix. The authors should explicitly mention this. I would also suggest to rephrase “once a diffusion model has converged to the optimum...”. At first I was confused about whether the authors were referring to convergence to the final sampling point or convergence of the parameters of the score network (I assume the authors are referring to the latter).

• The content of Section 6 concerns the study of score deviation from its ideal value. Also in this case, the authors do not to compare their results with the work by Karras, 2022, where it is stated (see their Section 5, page 9): “Inspecting the per-$\sigma$ loss after training (blue and orange curves) reveals that a significant reduction is possible only at intermediate noise levels; at very low levels, it is both difficult and irrelevant to discern the vanishingly small noise component, whereas at high levels the training targets are always dissimilar from the correct answer that approaches dataset average” How is this different from the claim in your article: “ the deviation between the learned denoising output and its optimal counterpart behaves differently in three successive regions...”?

• Finally, while the content of Section 6 is technically correct, I fail to see the geometrical perspective in the findings (is the l2 distance the only geometrical metric to consider?)

1. Mathematical writing is sloppy.
2. The observations are not sufficiently well justified, and their impact is limited.
3. Some of the mathematical results are trivial.

More specific remarks:

Section 2 :

• There is an ambiguity between distribution and density function. Given the formula of the score function used, for instance, in (2), $p_t$ denotes the probability density function with respect to the Lebesgue measure. Then, it is written that $p_0 = p_d$ is the empirical data distribution. This is unclear to me. The empirical data distribution does not admit a density wrt to the Lebesgue measure. I guess $p_d$ is not the empirical data distribution, but the density of the “theoretical” distribution of a data point.

• It is worth defining the precise meanings of $p_d$ and $p_n$ in the beginning of Section 2

• Eq 1 & 2, $dx$ should be replaced by $dx_t$

• Eq 3: this part is unclear for several reasons. First, it is not defined what is $\theta$, what is $\sigma$ and what is $r$? Second, the least squares problem should necessarily involve a minimization; it is not specified wrt to which quantity the minimization should be done. Third, what is the output of the least-squares problem and how it is related to the score?

• Proposition 1 can hardly be considered as a mathematical statement. Is it possible to formulate it in the conventional forme: let $r_{\theta}$ be … and $\tilde x$ be the output of a one-step Euler scheme …, then … ? Without exaggeration, with the current formulation I do not understand the claim of the proposition.

• From a mathematical point of view, the first claim of Prop 2 is not well formulated. Indeed, the quantity on the left-hand side is random, which means that the meaning of the convergence hidden in big-O notation should be made clear. Regarding the proof of this proposition presented in the appendix,

  • Eq 44 has no mathematical meaning. One cannot write $a = b \pm c$.
  • Eq 45 is wrong
    I believe what the authors try to do in this proposition can be done in a clean way using Talagrand’s Gaussian concentration inequality.

Section 3 :

• Observation 1 is poorly justified. The fact that the distance between the 10 points of the trajectory and the straight line connecting the first point to the last point of the set is smaller than 20 does not necessarily imply that the continuous-time path these points are sampled from is a line. It might very well be a spiral around a line or any other curve.
  There is a well known method in statistical data analysis for measuring how close a point cloud is to a line. It is called PCA and suggests to look at the spectrum of the correlation matrix. It would be more relevant to justify the closeness to a line by such a method, based on more than a dozen of points.
• Observation 2: it is not quite clear to me what the authors call “move monotonically”. Since the justification of observation 2 seems to be the blue curve of fig 2 (b), I guess the meaning of monotonicity here is to be understood as the monotonicity of the function that maps t to the distance between z(t) and z(0), where z is either the x or r.
• Observation 3, and Fig 3: I am not sure that the denoising trajectory converges faster than the sampling trajectory does. The speed of convergence is the derivative of the curve (the slope), which seems to be stronger for the sampling trajectory than for the denoising trajectory.

1. Lack of novelty. Most of the results/techniques have been observed/used in other papers.

• My main issue with this work is that it is unclear exactly how to use these theoretical insights and observations to improve diffusion models and diffusion model sampling. While it is interesting to link mean-shift algorithms to diffusion models it is not shown how this link leads to improved sampling. Similarly, while all the fast ODE solvers are shown to be related, it is unclear how to use this for better sampling.
• Theory is all quite simple (though useful), and is not particularly novel.
• I have seen the jump trick used in practice fairly often, and do not think this is a particularly novel insight. For example in RFDiffusion this trick is used [1]. Though I doubt this is the origin of this trick.

Overall I lean towards rejection, but could be convinced by a small amount of actionable insight, given the clear exposition and intuitive geometric figures. However, I note I do not have full knowledge of the novelty of this with regard to related work in this space.

[1] Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, David Baker. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. Science 2023.