Manifold Constraint Reduces Exposure Bias in Accelerated Diffusion Sampling

We sincerely thank you for your insightful reviews and valuable questions regarding the applicability of MCDO to high-order samplers and its effectiveness under large CFG scales. Below, we present detailed experimental results to address these points.

Q1: Applicability to High-Order Sampler

A7 To evaluate the performance of MCDO with a high-order sampler, we conducted experiments using DPM-Solver++ on SDXL with 5, 6, 10 steps. 60 samples were used for statistics estimation. The results, based on 10k generated images, are summarized below:

The results demonstrate that the proposed method consistently enhances performance when combined with DPM-Solver++. While the improvement of MCDO is more pronounced with DDIM compared to DPM-Solver++, this is consistent with the expectation that high-order solvers, such as DPM-Solver++, inherently reduce prediction errors. The results are also presented in Appendix A.10 of the revised manuscript.

Q2: Efficacy of MCDO with combined with larger CFG Scales

A8 To assess the efficacy of MCDO under varying CFG scales, we performed additional experiments using DDIM sampler on SDXL with larger CFG scales (8, 10 ,12) using 10 sampling steps. The number of samples used for statistics estimation is the same as that in the paper. The results are summarized below:

The findings demonstrate that MCDO significantly improves performance under large CFG scales, achieving a notable reduction in FID. This suggests that MCDO effectively alleviates the increased exposure bias introduced by higher CFG scales. The results are also presented in Appendix A.12 of the revised manuscript.

I highly appreciate the efforts of the authors in the rebuttal. And below is my feedback.

• Figs. 4 and 6 only employ 64 samples, which seems inconvincing. The authors seemed to miss this one.

• The clarification in W1.4 is not that convincing. I hope the authors could provide some experimental or theoretical results beyond the current response.

• Response in W1.5 is still wrong.

  1. First, there is a gap in the original Diffusion Model paper that, $\mathbf{x}_T$ in the forward diffusing process only approximates a Gaussian distribution with zero mean, i.e., $\mathbf{x}_T\approx\mathcal{N}(\alpha_T\mathbf{x}_0,\sigma_T^2\mathbf{I})$. Both theoretically and empirically, $\alpha_T$ is extremely small but still nonzero. Therefore, the conclusion of zero mean by induction at each $t<T$ is wrong in your rebuttal.
  2. Second, if the underlying data follows a distribution with extremely large mean, from my perspective, the zero-mean assumption is unreasonable and may lead to strong error in your pipeline. I hope the authors could conduct experiments on toy data to convince me, e.g., following setting of Reviewer Tp1G, the authors could apply your method on a Dirac Delta distribution $\delta(\mathbf{x}_0)$, in which $\mathbf{x}_0$ is extremely far away from the origin. Such distribution has large mean and zero variance, and its score function has analytical form.

W1.4 Different derivation with/without $\\| \\boldsymbol{x}_0 \\|$

A4 Thank you for your thorough and thoughtful review. We agree that the method you propose is effect, and we would like to offer further clarification to address some potential misunderstandings regarding our method:

1. We did not claim that it is essential to consider $\\boldsymbol{x}_0$.
2. Our proposal introduces an alternative approach that incorporates $\\boldsymbol{x}_0$ into the process.
3. We view both approaches (with and without $\\| \\boldsymbol{x}_0 \\|$) as practical (see Sections 4.2 and 4.3). The approach without $\\boldsymbol{x}_0$ can be interpreted as the objective of TS-DDIM [1].
4. However, on large resolution dataset (e.g., $1024\\times 1024$ for SDXL), we observe that our proposed method, which incorporates $\\boldsymbol{x}_0$, leads to improved results.

Hope this clarificaiton addresses your concerns. Please do not hesitate to let us know if further explanation is required.

[1] Li et al. Alleviating exposure bias in diffusion models through sampling with shifted time steps. ICLR 2024.

W1.5 Assumption on pixel mean of denoising observation $\\boldsymbol{\\hat{x}}\_0$

A5 Thanks for point this out. We would like to explain further.

1. Given that the diffusion model is trained to mininmize $\\| \\boldsymbol{\\epsilon}\_t - \\boldsymbol{\\epsilon}\_\theta^{(t)} \\|, \\boldsymbol{\\epsilon}\_t \\sim \\mathcal{N}(\\mathbf{0},\\boldsymbol{I})$. With a well-trained model, one has: $\\mathbb{E}[\\boldsymbol{\\epsilon}\_\\theta^{(t)}] \\rightarrow \\mathbb{E}[\\boldsymbol{\\epsilon}\_t] = 0$. For the first timestep in denoising process ($t=T$), $\\boldsymbol{\\hat{x}}\_t \\sim \\mathcal{N}(\\mathbf{0},\\boldsymbol{I})$. The expectation of pixel mean of $\\hat{\\boldsymbol{x}}\_0^{(t)}$ is $0$ as it is a linear composition of $\\boldsymbol{\\hat{x}}\_t$ and $\\boldsymbol{\\epsilon}\_\\theta^{(t)}$: $\\hat{\\boldsymbol{x}}\_0^{(t)} = \\frac{\\hat{\\boldsymbol{x}}\_t - \\sqrt{1 - \\bar{\\alpha}\_t} \, \\boldsymbol{\\epsilon}^{(t)}\_\\theta\\left(\\boldsymbol{x}\_t\\right)}{\\sqrt{\\bar{\\alpha}\_t}}$.
2. Considering that the noisy data prediction for $t-1$ timestep can be expressed as $\\hat{\\boldsymbol{x}}\_{t-1} = \\sqrt{\\bar{\\alpha}\_{t-1}} \\hat{\\boldsymbol{x}}\_0^{(t)} + \\sqrt{1 - \\bar{\\alpha}\_{t-1} - \\sigma\_t^2} \, \\boldsymbol{\\epsilon}\_\\theta^{(t)}\\left(\\mathbf{x}\_t\\right) + \\sigma\_t \\boldsymbol{\\epsilon}'\_t,\\quad \\boldsymbol{\\epsilon}'\_t \\sim \\mathcal{N}\\left(\\mathbf{0}, \\mathbf{I}\\right)$, we have: $\\mathbb{E}[\\boldsymbol{\\hat{x}}\_{t-1}] = 0$. Then according to the definition of denoising observation, the expectation of pixel mean for denoising observation at timestep $t-1$: $\\mathbb{E}[\\frac{1}{n}\\Sigma\_{i=0}^{n}\\hat{\\boldsymbol{x}}\_0^{(t-1)}] = 0$

W3: Visualization with photorealistic and obvious color shift

A6 We appreciate your observation and agree that further refinement of the correction strategy could potentially improve the sample quality. We acknowledge that some aspects related to the visualizatino presented in Figures 7 and 12 require clarification and would like to address your concerns in detail:

1. Higher saturation alone does not equate to higher image quality, and it should not be interpreted as a flaw unique to our method. While some color shifts (compared to the baseline which sometimes generates blurred and un-recognizable images) are noticeable, the higher saturation arises from our method's effectiveness in reducing models' prediciton error, therefore producing image with finer quality (sharper textures and richer structural details), as can be observed in all visualizations presented in paper.
2. As the sampling steps decreases, blurriness are observed in many baseline samples. As a consequence, it is reasonable for solutions which improve image quality make the generated results look "relatively saturated". Which is observed in many previous published works [2][3][4].
3. In Appendix A.13 of the revised manuscript, we include the visualization results generated using 1,000 steps which we believe will serve as a fair reference.

[2] Chen et al. On the Trajectory Regularity of ODE-based Diffusion Sampling, ICML 2024. [3] Si et al. FreeU: Free Lunch in Diffusion U-Net. CVPR 2024 Oral. [4] Lin et al. Common diffusion noise schedules and sample steps are flawed. WACV 2024.

Thank you for your insightful comments and questions. Please find our response below:

W1: Theoretical details in the paper

W1.1 Definition of $\\boldsymbol{x}\_0$ and $\\boldsymbol{\\epsilon}\_t$

A1 Thanks for pointing this out. $x_0$ is the clean image sampled from data distribution, while $\epsilon_t$ is the noise sampled from normal distribution. Since $e_t$ is independent of $x_0$, with a high enough latent dimension $n$ [1], one has: $\mathbb{E}[\| \boldsymbol{x}_0 + \boldsymbol{\epsilon}_t\|^2 ] = \mathbb{E}[\| \boldsymbol{x}_0\|^2] + \mathbb{E}[\|\boldsymbol{\epsilon}_t\|^2]$. This typo is corrected in the revised manuscript.

[1] Sven-Ake Wegner. Lecture notes on high-dimensional data, 2024.

W1.2: Relation between the Analytical Forms of $\\boldsymbol{\\hat{x}}_t$ and $\\boldsymbol{\\hat{x}}_0^{(t)}$ and Their Pixel Variance

A2 Thank you for pointing this out. We would like to provide further clarification. The analytical forms of $\\hat{x}\_0^{(t)}$ and $\\hat{x}\_t$ given in Equation 18 are presented to demonstrate the following:

1. During the generation process, the noisy data prediction $\\hat{\\boldsymbol{x}}\_t$ contains a component of scaled-down prediction error, denoted as $n\_t \\cdot \\boldsymbol{e}\_\\theta^{(t)}$, where $n\_t$ is the scale factor. Conversely, the denoising observation $\\hat{\\boldsymbol{x}}\_0^{(t)}$ incorporates a scaled-up prediction error, expressed as $d\_t \\cdot \\boldsymbol{e}\_\\theta^{(t)}$, where $d\_t$ is the scale factor.
2. The large magnitude of $d\_t$ suggests that $\\hat{x}\_0^{(t)}$ is predominantly influenced by the amplified noise prediction error term $d\_t \\cdot e\_\\theta^{(t)}$ at most timesteps. Thus, it is possible to correct the noise estimation error $\\boldsymbol{e}\_\\theta^{(t)}$ by incorporating information from the denoising observation $\\hat{x}_0^{(t)}$.

In other words, while the noisy data prediction $\\hat{x}\_t$ primarily reflects information about its ground truth noisy data $\\boldsymbol{x}\_t$, the denoising observation $\\hat{x}\_0^{(t)}$ conveys information about the prediction noise e_\\theta^{(t)}. This discrepancy leads to the following results:

1. Correcting the pixel variance of $\\hat{x}\_t$ using a scale factor will eventually scales the entire term, $\\boldsymbol{x}\_t + n\_t \\boldsymbol{e}\_\\theta^{(t)}$. Consequently, this leads to distortion of the $\\boldsymbol{x}\_t$ component in $\\hat{x}\_t$.
2. When correcting the pixel variance of $\\hat{x}\_0^{(t)}$ using the proposed method, one is correcting $\\boldsymbol{x}\_0 + d\_t \\boldsymbol{e}\_\\theta^{(t)}$. While errors in $\\boldsymbol{e}\_\\theta^{(t)}$ also influence the $\\boldsymbol{x}_0$ term, it is important to note that by recalculating the noise estimation with the corrected denoising observation (Equation 21) : $\\tilde{\\boldsymbol{\\epsilon}}^{(t)}\_\\theta = \\frac{x\_{t} - \\sqrt{\\bar{\\alpha}\_{t}} \\tilde{x}\_0^{(t)}}{\\sqrt{1 - \\bar{\\alpha}\_{t}}}$ the prediction error is reduced while the original $\\boldsymbol{x}\_t$ is preserved.

We hope this clarification addresses your concerns. Please feel free to reach out if further explanation is needed.

W1.3 Comparison between Equation 16 and Equation 19 regarding their role in reducing data-manifold distance.

A3: We respectfully disagree with your statement that "the authors conclude differently." It appears there may be some misunderstanding, and we would like to clarify this point further.

1. As clearly stated in Section 4.2, Equation (16) offers a manifold-based perspective to understand the effectiveness of the previous method, TS-DDIM [1], which relies on the assumption of "using pixel variance to estimate the sample distribution variance."
2. We agree with your observation that "minimizing the right-hand side in Eq. (16) could also lead to the reduction of the distance." As we view this approach as an effective explanation of the timestep-shifting strategy in TS-DDIM [1], from the manifold perspective. We acknowledge the simplicity and effectiveness of their proposed method, which inspired us to explore the concept of pixel variance further, and to leverage Equation 19 to develop the proposed method.

[1] Li et al. Alleviating exposure bias in diffusion models through sampling with shifted time steps. ICLR 2024.

Thank you for your hard work! I really appreciate it.

By the way, what do you think about the second point, strong assumption? I want to hear your (possibly informal) opinion about this.

Thank you for your hard work. Responding to six rebuttals must have been very challenging; I appreciate it.

I had two concerns: one was the FID hack, and the other was the strong assumption.

FID hack

Since the FID score is calculated by measuring the distributional distance between two sets of Inception-v3 features, it fundamentally differs from the pixel variance or L2-norm of an image.

I know your point. However, I am still curious. In my experience, the FID score is greatly influenced by the color distribution of images. In the ablation study, you applied MCDO from $T$ down to $t$, but couldn't my experiment be considered as applying MCDO only at $t = 1$? I didn't think it was a baseless experiment. If I'm wrong, please correct me.

In addition to the improvements in FID score, our approach consistently achieves better results across other metrics, such as CLIP score, IS score, and sFID.

I agree.

We agree with your opinion that using the statistics collected from the training data could potentially lead to even better results.

I had misunderstood the method. I'm sorry, and thank you for clarifying.

Strong Assumption

Firstly, we introduce manifold assumptions extended or borrowed from previous works [1][2].

I kindly disagree with this statement because [1] and [2] do not use mean and variance information to approximate the manifold. MCG utilizes the property that the input gradient of the diffusion model $\nabla_{x_t} \epsilon(\cdot)$ is aligned with the manifold. This is also the case in DPS [4]. MPGD approximates the manifold using a VAE. That is, while they begin their theoretical development by asserting from the Gaussian annulus theorem that $x_t$ has a low-dimensional manifold structure, they do not claim that this can be approximated using mean and variance.

Personally, the reason I cannot readily give a positive evaluation of this work is that I find it hard to accept that resolving exposure bias through mean and variance should be considered a Manifold Constraint. Generally, the tangent vectors of the latent space manifold have clean, interpretable signals, as seen in the guidance vectors of DPS and MCG. This is even more evident in [5, Figure 6]. However, since the Gaussian annulus through mean and variance does not possess such manifold structure, the claim that improvements are due to the diffusion model manifold constraint sounds somewhat like overclaiming to me. I question whether the Manifold Constraint is an appropriate mathematical framework to excellently explain your method.

Or... it could just be me thinking this way. I would not consider this point as crucial one for evaluation.

[4] Chung et al. Diffusion Posterior Sampling for General Noisy Inverse Problems

[5] Park et al. Understanding the Latent Space of Diffusion Models through the Lens of Riemannian Geometry

To this end, some of my concerns regarding the FID hack have been resolved. However, it’s still difficult to be fully convinced because I haven’t seen results where the mean/variance regularization is applied only to the t=1 (or t=0). As for the strong assumption, I find it difficult to agree with the authors. Therefore, I will maintain my score for now. If the FID experiment yields positive results, I am considering raising my score to a 6.

Q2: Results with other solvers & results on DDIM with NFE = 50

A3 Thank you for your suggestion. We summarize the results with DPM++ sampler as follows:

These resuls indicate that our method enhances the performance of high-order solver: DPM-Solver++. Given that DPM-Solver++ inherently reduces prediction error and causes less exposure bias at low NFE, the improvements of our method are slightly smaller compared to first-order solvers (e.g. a 2.57 FID decrease for DDIM). The results are also presented in Appendix A.10 of the revised manuscript.

We conducted experiments with DDIM using NFE = 50 on the SDXL dataset but did not observe significant improvements in the quantitative results with the current hyperparameters ($S=20$, $t_{\text{thre}}=100$). This suggests that, at higher NFE values, where step sizes are smaller, diffusion models likely make fewer prediction errors, and consequently, the exposure bias problem discussed in the paper becomes less severe.

However, it is important to note that our method is specifically designed for accelerated sampling in diffusion models, with a primary focus on low NFE scenarios. While it is true that increasing NFE can improve the results to some extent, we believe that this does not diminish the core value of our approach. The main benefit of our method lies in its ability to address exposure bias in scenarios with fewer sampling steps, which is the key challenge in accelerated sampling.

We greatly appreciate your recognition of our method’s compatibility with previous work. Below, we provide answers to your concerns.

W1: Assumption in variance-manifold relation

A1 Thank you for your suggestion. We agree that it is not sufficient to fully understand data manifold with the variance concept. However, we would like to clarify that we do not claim that "manifold can be understood solely by considering variance." To better explain our method, we summarize the main ideas of our paper as follows:

1. Firstly, we introduce manifold assumptions extended or borrowed from previous works [1] [2].
2. Then, we establish a connection between the data-to-manifold distance and data pixel variance, by assuming: $\\mathbb{E}[\\boldsymbol{\\epsilon}\_\theta^t]=0, \\mathbb{E}[\\boldsymbol{x}\_t]=0$ . These assumption are justified as follows:
   1. The noise $\\boldsymbol{\\epsilon}\_t$ added to clean data during the training phase follows normal distribution: $\\boldsymbol{\\epsilon}\_t \\sim \\mathcal{N}(\\mathbf{0}, \\boldsymbol{I})$, which implies $\\mathbb{E}[\\boldsymbol{\\epsilon}\_t]=0$. Consequently, assuming $\\mathbb{E}[\\boldsymbol{\\epsilon}^t\_\\theta]=0$ is reasonable for well-trained diffusion models.
   2. The predicted noisy data at timestep $t-1$ can be expressed as $\\hat{x}\_{t-1} = \\sqrt{\\bar{\\alpha}\_{t-1}} \\frac{x\_t - \\sqrt{1 - \\bar{\\alpha}\_t} \, \\boldsymbol{\\epsilon}^{(t)}\_\\theta\\left(x\_t\\right)}{\\sqrt{\\bar{\\alpha}\_t}},+ \\sqrt{1 - \\bar{\\alpha}\_{t-1} - \\sigma\_t^2} \\, \\boldsymbol{\\epsilon}\_\\theta^{(t)}\\left(\\mathbf{x}\_t\\right) + \\sigma\_t \\boldsymbol{\\epsilon}'\_t,\\quad \\boldsymbol{\\epsilon}'\_t \\sim \\mathcal{N}\\left(\\mathbf{0}, \\mathbf{I}\\right).$ Then, as long as $\\mathbb{E}[\\frac{1}{n}\\Sigma\_{i=0}^n\\boldsymbol{x}\_{t\_i}] =0$, one has $\\mathbb{E}[\\frac{1}{n}\\Sigma\_{i=0}^n\\boldsymbol{x}\_{{t-1}\_i}] =0$. Considering that the initial noise is sampled from normal distribution: $\\boldsymbol{x}\_T \\sim \\mathcal{N}(\\mathbf{0},\\boldsymbol{I})$, we have $\\mathbb{E}[\\frac{1}{n}\\Sigma\_{i=0}^n\\hat{\\boldsymbol{x}}\_{t-1}]=0, \\forall t \\in [1, T]$.
3. Leveraging the pixel variance-manifold relation, we achieve the following:
   1. provide a new perspective on the effectiveness of prior work [3] from the viewpoint of the data manifold.
   2. introduce a training-free and tuning-free method, whose effectiveness is demonstrated across various models and datasets.

We hope this explanation addresses your concerns and clarifies any potential misunderstandings. Please do not hesitate to let us know if further clarification or additional details are required.

Q1: "FID Hack" & using $q_0$ to Implement MCDO

A2 We do not consider the proposed method as a "hack" of the FID score. We explain our approach as follows:

1. The primary objective of our method is to improve model's performance under low NFE situaiton. By guiding the deviated data toward their manifold, we achieved in reducing the exposure bias during generation. To achieve this, we leverage pixel variance estimation collected from generation process.
2. Since the FID score is calculated by measuring the distributional distance between two sets of Inception-v3 features, it fundamentally differs from the pixel variance or $L_2$-norm of an image. For instance, modifying the pixel variance of an image does not necessarily improve its quality.
3. In addition to the improvements in FID score, our approach consistently achieves better results across other metrics, such as CLIP score, IS score, and sFID. This suggests that the effectiveness of our method stems from its ability to reduce the prediction errors made by the models, rather than manipulating any metric.

We agree with your opinion that using the statistics collected from the training data, $q_0$, could potentially lead to even better results. Given that in this case, the model would have access to the real training data, directly modifying the generated data distribution to "hack" FID, without improving its prediction accuracy.

[1] Chung et al. Improving diffusion models for inverse problems using manifold constraints. NeurIPS 2022.

[2] He et al. Manifold preserving guided diffusion. ICML 2024.

[3] Li et al. Alleviating exposure bias in diffusion models through sampling with shifted time steps. ICLR 2024.

We thank reviewer fuNA for the acknowledgement of our contributions and the helpful comments. Below, we provided a point-to-point response to all comments.

W1: Explanation for denoising observation

A1 We appreciate it that you point this out. In this paper, we leverage the term denoising observation from paper DDIM [1], which proposes the widly used accelerated sampling technique.

W2: Discussion on the Pre-computation of the Full Denoising Sequence

A2 Thank you for highlighting the trade-off between the number of steps in pre-computation and the performance of the method. In the paper, we use a long denoising sequence (1,000 steps for SDXL experiments on the MS-COCO dataset) to better capture the statistics required by the proposed method. However, denoising trajectories with fewer timesteps are also feasible. For instance, in experiments on the CelebA-HQ dataset, we collect statistics using denoising sequences of 500 timesteps, and the proposed method still demonstrates noticeable improvement with statistics derived from a shorter sampling trajectory.

Nevertheless, we believe that, in general, increasing the number of sampling steps (and thus reducing the sampling intervals) leads to more accurate estimation of the pixel variance of $\hat{x}_0^{(t)}$.

W3: Diversity of Reference Samples and Details in the Collection

A3 To ensure a sufficient level of diversity in the samples collected for $v_t$ estimation:

1. For text-to-image generation with MCDO, we randomly select 20 distinct prompts from the dataset to guide the generation of reference samples (one sample per prompt).
2. In class-conditional generation on ImageNet, 64 randomly sampled class IDs were used to guide the generation of 64 images (one sample per class).

We appreciate your insights into the trade-offs between performance and the number of reference samples. For ease of implementation, we use $S=20$ for text-to-image generation with SDXL, and $S=64$ for the other experiments. While we agree that the $v_t$ estimation could potentially benefit from an increased number of reference samples, considering the improvements already achieved by MCDO in these experiments, we believe that the current choice of hyperparameters is practical. Therefore, we have decided not to further tune them for potentially better quantitative results.

[1] Song et al, Denoising Diffusion Implicit Models, ICLR 2021.

I appreciate the authors' efforts in rebuttal. I decide to keep my positive rating. Thank you.

Thank you for your insightful feedback.Our answers to each question are shown below:

W1 Explanation of Assumption 3

A1 Thanks for raising this question about Assumption 3. We agree that for most of timesteps, especially those in the early denoising stage, $\hat{x}_0$ have a very different distribution from that of $x_0$. However, this difference is a fundamental aspect of our method’s design, which is detailed in Algorithm 1.

The objective of MCDO is to dynamically adjust $\hat{x}_0$, bringing it closer to its corresponding distribution based on the current timestep $t$, rather than adhering to the fixed distribution of $x_0$. For example, if the distribution of $x_0$ were used to correct that of $\hat{x}_0^{(t)}$ at the initial timestep (t=T), the entire process would be adversely affected.

Q1 Fewer steps have lower FID on ImageNet

A2 Thank you for pointing this out. We also observed this phenomenon during our experiments and, as a result, re-evaluated all methods using a new environment. The outcomes of this re-evaluation were consistent with the results presented in the paper.

As shown in the visualization of the generated images provided in Appendix A.8, while fewer steps yield lower FID values, the corresponding image quality is noticeably worse, which aligns with general intuition.

Since FID is calculated based on the distance between estimated distributions of two Inception-v3 features, it may not always perfectly reflect image quality in all scenarios. We believe this could explain the observed behavior on this dataset.

Q2 Explanation of Equation 11 and Equation 16

A3 Equation 11 is based on the triangle inequality. As Equation 16 is derived by plugging $\| \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon_t \| \approx \sqrt{\| \sqrt{\bar{\alpha}_t} x_0 \|^2 + \| \sqrt{1-\bar{\alpha}_t} \boldsymbol{\epsilon}_t \|^2}$ into $d(\hat{x}_t,1, \mathcal{M}_t) \gtrapprox \left| \| \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon}_t \| - \sqrt{n\text{Var}(\hat{x}_t)} \right|$. Typos in Equation16 are corrected in the revised manuscript.

Q3 Explanation of approximation in L261

A4 To better explain L261, we would like to refer you to the Equation 14 in L258-259. Equation 14 says that, for random data sampled from n-dimension normal distribution: $\epsilon \sim \mathcal{N}(0,1,\mathbb{R}^n)$, the expectation of its norm satisfy: $|\\mathbb{E}[\\|\\boldsymbol{\\epsilon}\\|] - \\sqrt{n} | \\leq \\frac{1}{\\sqrt{n}}$ [1]. As $n$, the latent space dimension in diffusion models is usually large in diffusion models (e.g. $n=12288$ for LDM-4 trained on CelebA-HQ), we have: $\\frac{1}{\\sqrt{n}} \\rightarrow 0$. Therefore, $| \\mathbb{E}[\\|\\boldsymbol{\\epsilon}\\|] - \\sqrt{n} | \\| \\rightarrow 0$. That is to say, with high probability, we have $\\|\\boldsymbol{\\epsilon}\\| \\approx \\sqrt{n}$.

Alternatively, with large $n$, one can directly consider this: $\\mathbb{E}(\\|\\boldsymbol{\\epsilon}\\|^2) = \\mathbb{E}(\\Sigma\_{i=0}^{n}\\boldsymbol{\\epsilon}\_i^2)= \\Sigma\_{i=0}^{n}\\mathbb{E}(\\boldsymbol{\\epsilon}\_i^2) = \\Sigma\_{i=0}^{n}\\left(\\mathbb{E}(\\boldsymbol{\\epsilon}\_i)^2+\\text{Var}\\left(\\boldsymbol{\\epsilon}\_i\\right)\\right)= \\Sigma\_{i=0}^{n}\\left(\\text{Var}\\left(\\boldsymbol{\\epsilon}\_i\\right)\\right) =n$. Which leads to $\\mathbb{E}[\\|\\boldsymbol{\\epsilon}\\|] \\approx \\sqrt{n}$.

[1] Sven-Ake Wegner. Lecture notes on high-dimensional data, 2024.

Q4: Explanation of L271 and generalization of observation presented in figure 3

A5 Thanks for raising this question. In fact, the objective of Equation 16 is to elucidate the relation between $d(x_t,1,\mathcal{M}_t)$ and the deviation of $\text{Var}(x_t)$. As this perspective can provide a manifold perspective to understand the effectiveness of previous published work TS-DDIM [2], which is based on assumption of using pixel variance of a single image to estimate distribution variance.

We appreciate your insightful observation that a decrease in pixel variance could potentially make $d(x_t, 1, \mathcal{M}_t)$ larger than $r_t$. However, we do not claim that the pixel variance of $x_t$ will necessarily decrease with fewer NFEs in all cases, as this would be too strong of an assumption, even though such observation might holds true statistically. Again, thank you for pointing this out.

Q5 Saturated examples

A6 Thanks for mentioning this. We would like to answer this question in detail below:

1. As the sampling steps decreases, blurriness are observed in many baseline samples. Therefore, it is reasonable for solutions which improve image quality make the generated results look "relatively saturated". Which is also observed in experiments in previous published work [3].
2. We contribute the higher saturation to our method's effectiveness in reducing models' prediciton error, producing image with finer quality (sharper textures and richer structural details), as can be observed in all visualizations presented in paper.
3. To provide a fair illustration of the saturation level, we have included qualitative results generated with 1,000 steps in Appendix A.13 of the revised manuscript. These results demonstrate that our method achieves a saturation level similar to that of high-quality images generated with 1,000 steps, when compared to the images produced without our method.

[2] Li et al. Alleviating Exposure Bias in Diffusion Models through Sampling with Shifted Time Steps. ICLR 2024. [3] Chen et al. On the Trajectory Regularity of ODE-based Diffusion Sampling, ICML 2024

Thank you very much for your helpful reviews and feedback, we now reply to your concerns:

W1&Q2&Q3: Assumptions on the mean value of noise estimation and noisy data, and refinement for Eq. 15

A1 Thank you for pointing this out. We explain each as follows.

1. In the proposed method, the assumption that the expectation of the pixel sum of the noise estimation and the noisy data is zero serves as tool to relate the pixel variance of the noisy data and its deviation to the manifold.
2. Relationship between these two concepts provides a novel insight into the methodology of TS-DDIM, and forms the basis for developing our proposed method.
3. We made these assumptions based on statistic facts listed as follows:
   1. Denoting the noise added to clean data during training phase as $\\boldsymbol{\\epsilon}_t$. As it follows normal distribution: $\\boldsymbol{\\epsilon}_t \\sim \\mathcal{N}(\\mathbf{0}, \\boldsymbol{I})$, we have $\\mathbb{E}[\\boldsymbol{\\epsilon}_t]=0$. Therefore, when diffusion models are well-trained, assuming $\\mathbb{E}[\\boldsymbol{\\epsilon}^{t}\_\theta]=0$ is reasonable.
   2. Given that the predicted noisy data of timestep $t-1$ can be expressed as $\\hat{x}\_{t-1} = \\sqrt{\\bar{\\alpha}\_{t-1}} \\frac{x\_t - \\sqrt{1 - \\bar{\\alpha}\_t} \, \\boldsymbol{\\epsilon}^{(t)}\_\\theta\\left(x\_t\\right)}{\\sqrt{\\bar{\\alpha}\_t}},+ \\sqrt{1 - \\bar{\\alpha}\_{t-1} - \\sigma\_t^2} \\, \\boldsymbol{\\epsilon}\_\\theta^{(t)}\\left(\\mathbf{x}\_t\\right) + \\sigma\_t \\boldsymbol{\\epsilon}'\_t,\\quad \\boldsymbol{\\epsilon}'\_t \\sim \\mathcal{N}\\left(\\mathbf{0}, \\mathbf{I}\\right).$ Then, as long as $\\mathbb{E}[\\frac{1}{n}\\Sigma_{i=0}^n\\boldsymbol{x}\_{t\_i}] =0$, one has $\\mathbb{E}[\\frac{1}{n}\\Sigma\_{i=0}^n\\boldsymbol{x}\_{{t-1}\_i}] =0$. Considering that the initial noise is sampled from normal distribution: $\\boldsymbol{x}\_T \\sim \\mathcal{N}(\\mathbf{0},\\boldsymbol{I})$, we have $\\mathbb{E}[\\frac{1}{n}\\Sigma\_{i=0}^n\\hat{\\boldsymbol{x}}\_{{t-1}\_i}]=0, \\forall t \\in [1, T]$.
4. In addition, extensive experiments demonstrate the effectiveness of our approach across various model and datasets. Therefore, we believe these assumptions are acceptable.
5. Nevertheless, we agree that these assumptions may not hold for every case. In some application, there might exist cases where the mean of latent does not approaches zero, we leave this as further investigation in future work.
6. Thanks for your suggestion to improve Equation 15. We correct the problem you mentioned in the revised manuscript.