On the Relation Between Linear Diffusion and Power Iteration

Thank you for your review. Following the general remark, we have added an explicit mathematical explanation of our statement in the high noise regime, to justify the convergence of the total projection iterator.

Answering your specific questions:

• The low dimensional subspace prior is very common considering natural images, and was considered in many previous works - in the context of diffusion models (which we already cited in the main text) and also more generally in computer vision - we added references to further justify this model. Of course, more realistic datasets entail more information, and encompass a larger subspace, which may be a collection of different smaller subspaces itself.
• We added a short paragraph regarding the power iteration method in the introduction of the main text, with further details in an appendix for the ease of reading.
• We added a clarification regarding the notations used throughout the paper, in addition to the definition of $u_i^t$.

I thank the authors for addressing my concerns. I will keep my positive score.

Thank you for your review. The main point our proof is that as long as the elements on the diagonal are ordered, the total product converges to the first eigenvector. We believe that his issue answers both of your major questions. We elaborate on that next:

(i) It is true that all the eigenvectors go to zero, but since we are interested in the direction of the final vector (and not its norm), the final sample is aligned to the first eigenvector, where the ratio between the largest and second largest eigenvalues determines the rate of convergence. This is exactly the same as the convergence process in the classic power iteration method, which we added in an appendix for the ease of reading. If we also want to consider the norm of the final vector, the first eigenvector will always be the dominant part, no matter how small the total norm is. In addition, Figure 5 explicitly shows the convergence to the first eigenvector (and specifically not to zero). Following your comment and to make it clearer, we added more details to the proof in the main text, including a closed form expression that covers the large noise regime. We hope it is easier to follow now.

(ii) In accordance with the main idea behind our proof, the resemblance to the linear case lies in the fact that the rate of decay depends on the index. It makes sense that the overall rates are faster in the nonlinear model compared to the linear case, since the nonlinear model attends to more complicated datasets. Following your comment, we have added this discussion to the main text, and modified the scale of Figure 7 in order to also address your minor question as well.

Note that we have updated manuscript to address all the concerns. Changes are marked in blue. Please let us know if there any remaining issues and we would be happy to answer. If not, we would appreciate if you would kindly consider updating the score.

Thanks for your reply. However, my questions are not well addressed.

(i) First, I have doubt on whether it is rigorous to express $U_{\tau_{i}+1}$ as $\text{diag}[1,...,\frac{c_i}{c_{0}}, \mathbb{A}]$ (line 394-399). As mentioned by the authors themselves (line 338-339), this nice form depends on the assumption of small angles, which only holds when $t\rightarrow 0$. Can the authors justify why they can derive the equation on lines 394-399 based on this assumption while it does not hold anymore? This is a very important question since the whole argument that linear diffusion model generates samples that align only with the first eigenvector explicitly depends on this closed form expression.

(ii) Furthermore, under this closed form expression , we have $\mathbb{E}x_gx_g^T=U_0\text{diag}[c_0^{2T}/T, c_1^{2T}/T,...]U_0^T$. Since all the $c$'s are within the range [-1,1] (they are cosine angles), the norm of the final generated images will diminish to 0 as your sampling steps increase. This is very inconsistent with what's happening in practice, even under the linear diffusion assumption-increasing the number of sampling steps will increase the generation quality, not pushing the samples to zero norm. Therefore, this serves as another evidence that the assumptions are not valid and consequentially the resulting does not accurately align with practice (even for linear diffusion). This inconsistency with practice can be clearly observed from Figure 6, where the authors perform numerical experiments based on empirical covariance. According to the theory, as T keeps increasing, the generated images should have increasingly smaller norm and at the same time become more and more concentrated on the first eigenvector. However, from Figure 6 we observe that the norms of final generation do not show decreasing trend. More importantly, the generated images do not concentrate only on the first eigenvector, instead, for large T, the generated samples have significant energy on the first 4 eigenvectors, not just the first. I doubt if keep increasing T will further prompt the generation to concentrate on the first eigenvector since from current figure the final generation seem to be identical (become stable) after T=110.

(iii) I am wondering why don't you theoretically study the case that the population covariance is given, i.e., the case for Figure 6, where we don't need to make assumption 4.2 and assumption 4.3 and the reverse sampling trajectory $x_t$ will have a simple close form solution due to the linearity. It is worth studying how the linear diffusion models work under this scenario. And maybe you can then train a linear diffusion models to see the differences between the case that population covariance is given.

(iv) Lastly, the empirical experiments in Figure 7 are not convincing enough. For section 5, if I understand correctly, the authors want to show that the convergence for lower index (darker color) should be faster. However, the current experiments do not validate this sufficiently. As shown in Figure 7, (i) the angles of some darker curves go to zero slower than the brighter curves (ii) a lot of the very bright (yellow) curves overlap with the very dark curve, showing similar convergence rate. Therefore, the Figure shows phenomenons that contradict with the main statement of the paper.

Thanks for your response, but my questions are not addressed.

(i) I understand $\mathbb{A}$ is not diagonal, I abuse the notation here a little bit. But my main concern is why you can write the diagonal entries of $U^T_{\tau_i}U_{\tau_{i+1}}$ as $\frac{c_i}{c_0}$ when the angle is not small? Please explain this.

(ii) I understand Figure 6 is normalized and this figure contradicts with your theorem in the following ways:

• First, according to the theory the $x_t$ should concentrate on the first eigenvector as T increase. However, the Figure 6 shows there are significant energy on the first 4 eigenvectors, not just the first.

• Second, from the plot I doubt whether keep increasing T will push $x_g$ to further concentrate on the first eigenvector since it is clear after T=110, $x_g$ is not changing anymore, achieving a stable solution.

(iii) Since currently Figure 6 does not align with theorem 4.1, I suggest directly studying what's happening in Figure 6, which should be much easier since you can easily derive the close form expression for the denoising chain $x_t$ without any extra assumption (just combining equations (10), (11) and (13) ). Does this close form denoising chain align with your theorem 4.1?

It is confusing to me why you make assumptions 4.1 and 4.2 if you believe Figure 6 aligns with your theorem (I don't think they well align). Why don't you directly study the property of denoising chain by considering the empirical covariance.

(iv) I have no misunderstanding on section 5 and Figure 7. What I mean darker curves should converge faster is that given a $t>0$, we should see the sine angle of the darker curve is smaller than that of a brighter curve, i.e., the low index eigenvectors of $U_t$ should converge to $U_0$ earlier.

Again, Figure 7 contradicts with your main statement in the following ways:

• First, a lot of the very dark curves (low index eigenvectors) have higher slope compared to the less dark curves, contradicting with the statement that lower index should have lower slope.

• Second, a lot of the very dark curves overlap with the very bright curves (a lot of dark curves seem to be hided behind the bright curves), contradicting with the statement that lower index should have lower slope.

Overall, there is a huge gap between section 5 and the previous sections.

Thanks for your reply, however, my questions remain unresolved.

(i ) First, I maintain my doubt on the assumptions. Second, I don't understand the meaning of the following statement: "Our takeaway from Section 5 is that (high complexity) nonlinear diffusion models consists of many low scale linearized environments, and can navigate a diverse set of linearized regions during the generation process (as illustrated in Figure 7)". The statement is quite abstract. Please make your definitions more concrete:

• Please clearly define the meaning of "local scale linearized environement", "navigate".
• Please elaborate how the empirical results lead to your conclusion. Right now, the reasoning is not adequate.

(ii) Third, could you please explain why $\Sigma_t+\sigma_t I$ and $\Sigma_t$ are not simultaneously diagonizable? They can still be expressed using $\Sigma_0$, right?

##########################################################

(iii) Next I want to discuss more on the denoising chain:

• You mentioned in line 128-129 that "we choose to present the standard diffusion models...". The standard way of training diffusion models is to learn the score of intermediate noisy distributions, which is exactly equivalent to the denoising problem I formulated in my last comment, not the sequentially denoising problem you wrote. Furthermore, in line 151-154 you wrote that you are learning the denoiser that project the noisy samples on to the "target" distribution, which also suggests you are learning the denoiser that predicts clean image from the noisy image. Could you please clarify this? If you are considering the denoiser that predicts the next noise scale from a higher noise, then it is not the standard diffusion model, making the paper of less interest. In Ho et.al (2020), the $\epsilon_\theta(x_t,t)$ in equation (6) represents the estimated $x_t-x_0$, not $x_t-x_{t-1}$. You should write your denoising objective explicitly in the paper so that readers can notice that it is not the standard diffusion model.

• My derivations in my previous comment demonstrate that linear diffusion models with standard reverse sampling process generate samples that follow a Gaussian distribution, which contradicts with your theorem. You mentioned that you still observe the concentration on the first eigenvector by analyzing the denoising chain. This inconsistency implies your denoising chain cannot recover the clean distribution with the score of intermediate distributions. Therefore, from my point of view, I don't find it meaningful to study this denoising chain unless it can recover the clean data distribution. Moreover, I am confused why the authors do not study the standard diffusion model with standard reverse sampling process. I don't see any significant difficulty in doing so.

• Please modify line 500-505 since standard linear diffusion models are equivalent to sample from a Gaussian distribution, which can generate diverse samples without injected noise. Or you may keep the current conclusion but explicitly indicate the differences between the standard diffusion and the special diffusion you are studying (I am not sure whether it can be called as 'diffusion model'). Otherwise, the conclusion can be misleading.

• Another related issue is that you cannot assume $\sigma_t=\frac{1}{T}$ (line 354). If you choose this schedule, then $\epsilon_T \sim \mathcal{N}(0,\frac{1}{T})$, which implies more sampling steps result in smaller amount of noise at the end stage of diffusion forward process. However, the noise level at the end of the forward diffusion should be sufficiently large so that the noisy distribution at time step $T$ is approximately a Gaussian distribution with zero mean. If this is violated, the reverse sampling process starting from a random Gaussian noise can not recover the clean distribution.

##################################################################

After seeing the authors' last comment, I have more questions and become more confused on this work, particularly in the differences between this work and the standard diffusion models. I will maintain my score for now. I suggest the authors make major revision to the paper before submitting it again.

Thank you for your review. We will first clarify the meaning of the results in Section 5 (per your points (2) and (3)), and then address the gap compared to nonlinear deterministic samplers (point (1)).

The results in section 5 (e.g. Figure 7) do not try to describe the distribution of the generated data - each plot is calculated per a single generated example. They are intended to show that in close proximity to generated samples, the denoiser base, i.e., the linearization of the nonlinear network, resembles the behavior of the linear denoising chain (i.e., the results in Figure 2). At t=0, the network Jacobian is aligned to the generated sample $x_0$, as was shown in previous work (Kadkhodaie et al., 2024). When t grows larger, the angle w.r.t. the base in t=0 grows larger, in a rate proportional to the index, like in the linear denoising chain. Thus, this section is not intended to show the convergence of a real network to the entire spectrum, but to show the resemblance of its local structure to the linear case we analyze. We have added this clarification to the main text.

Now, for the gap between the linear and nonlinear deterministic samplers (point (1)). This analysis focuses on the local behavior of the nonlinear denoiser at the end of the generation process, demonstrating its similarity to a linear denoising chain. Each plot represents a single generation path, not the overall distribution of generated outputs. While linear diffusion models are easy to analyze, they may struggle to generate complex datasets. Nonlinear models, on the other hand, can navigate a diverse set of linearized regions during the generation process (as illustrated in Figure 7). This allows them to generate diverse data even without added noise, unlike linear models which ultimately converge in mean to a single point (Theorem 3.4) and therefore require noise injection for diverse outputs. This contrasts with some deterministic nonlinear samplers (e.g., DDIMDMP) that do not rely on added noise. We have added this discussion to the main text. We chose to show how the final generated distribution depends on the choice of parameters, where one can control the mean of the generated spectrum (this might be a feature for some applications, such as segmentation via diffusion, etc.). Indeed, it might be interesting to derive the optimal parametrization for the convergence of the linear model - we leave this for future work, as it not the main focus of the paper.

Answers to Minor Claims/Questions:

(1) what are $x_j$ and $n$ in eq. (26)?

$x_j$ is a generated sample, out of $n$ examples. We added this definition to the main text.

(2) should $v_j$ right after eq. (26) be $u_j$?

That is correct, thank you. We fixed it in the main text.

(3) line 487: what is $J$ in $sin(\theta_J)$? is it $i$?

J stands for Jacobian, as opposed to $\theta_{PCA}$ in the linear model. We follow the common practice of using the Jacobian of a non-linear model as its local linearization. We have clarified this in the main text.

Note that we have updated manuscript to address all the concerns. Changes are marked in blue. Please let us know if there any remaining issues and we would be happy to answer. If not, we would appreciate if you would kindly consider updating the score.