Thank you for your valuable comments and suggestions. We provide our response to each question below.

Weakness 1: The universe error analysis for general $\tau$ and $\beta_t$.

In this part, we first provide a universe complexity for general $\tau \in [1,+\infty)$ and $\beta_t\in [1,t^2]$ under the reverse SDE, which covers current diffusion models. Then, we discuss the influence of different $\tau$ and $\beta_t$ in detail.

where

(a) The general formula covers current diffusion models.

In this paragraph, we show that our general formula can cover current models and provide the sample complexity. The key part of the sample complexity is balancing the first two terms: the reverse beginning and the discretization term. When $\beta_t=1$ and $\tau=1$, the drifted VESDE becomes VPSDE and $m_t=\exp{(-T)}$, which leads to a logarithmic $T$ and would not influence discretization term heavily. Then, we achieve $\tilde{O}(1/\epsilon\_{W_2}^8\epsilon\_{TV}^2)$, which has the same order with [1]. When $\beta_t=1$ and $\tau =T$, our formula is similar but slightly better (as shown in Figure 2 and our real-world experiments) to pure VESDE (\sigma_t^2= t) and we can achieve $\tilde{O}(1/\epsilon\_{W_2}^8\epsilon\_{TV}^6)$ results ([2] achieve slightly better results $1/\epsilon\_{W_2}^8\epsilon\_{TV}^4$ since they assume strong LSI assumption). When considering $\beta_t=t$ and $\tau =T^2$, the general formula is similar to pure SOTA VESDE (\sigma_t^2=t^2) and achieve the first polynomial sample complexity $\tilde{O}(1/\epsilon\_{W_2}^8\epsilon\_{TV}^6)$ under the manifold hypothesis. We also note that the above results holds for pure VESDE with $\sigma_t^2= t$ and $t^2$.

(b) When given a fixed $\beta_t$, the optimal $\tau$ has the same order with $\beta_T$.

As shown in (a), pure VESDE has a worse $\epsilon_{TV}$ dependence compared to VPSDE, which comes from large reverse beginning terms (the first term). For example, when considering $\beta_t=t$ and $\tau =T^2$, $m\_T= e^{-1/2}$ and $\sigma\_T^2=(1-e^{-1})T$, which leads a polynomial $T$ and heavily influence the second discretization term. Hence, the optimal choice of $\tau=T$ instead of $T^2$. With $\tau = T$, $m\_T=\exp{(-T)}$ and the complexity is $\tilde{O}(1/\epsilon\_{W_2}^8\epsilon\_{TV}^2)$ (Thm. 5.2), which has the same with VPSDE (this result also shows that the choice of VPSDE is optimal.). For $\beta_t=t^2$, the optimal $\tau$ is $T^2$, which has the same order with $\beta_T$.

We will make the above discussion clearer in the next version.

Weakness 2: The real-world experiments on CelebA 256.

We do experiments on the CelebA 256 dataset (a common face dataset) and show that our drifted VESDE can improve the results of pure VESDE without training from the quantitative and qualitative perspectives. Please see the experiment detail, discussion, and generated images in the global rebuttal part.

Q1: The general $W_p$ distance.

For the reverse SDE setting, similar to Corollary 5 of [1], by using the projection technique, we can achieve pure $W_2$ guarantee $\tilde{O}(1/\epsilon\_{W\_2}^{12})$, which has the same order with [1].

For the reverse PFODE setting, similar to the tangent-based method for VPSDE [3], our results can extend to $W_p$ for any $p\ge 1$. We will make it clearer in the next version.

Q2: The choice of $\tau$.

Thanks for the helpful comment on our general formula. As shown in Weakness 1, our general formula can consider $\tau\in [1,+\infty)$ and achieve polynomial sample complexity under the reverse SDE. We also show that our formula can recover current VPSDE and pure VESDE models and go beyond with $\tau \in [1,T^2]$.

For the reverse PFODE, as shown in Lem. 6.3, if considering $\tau =1$ (the VPSDE setting), there would be an additional $\exp{(T)}$ term. To avoid this term, we need to use the variance exploding property of VESDE. Hence, we choose $\tau\in [T,T^2]$, which represents two common VESDE choices under reverse PFODE (We note that Thm. 6.2 and Coro. 6.3 also hold for pure VESDE.)

We will make it clearer in the next version.

[1] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2022). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215.

[2] Lee, H., Lu, J., & Tan, Y. (2022). Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35, 22870-22882.

[3] De Bortoli, V. (2022). Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.05314.

Thanks again for your insightful suggestions and comments! According to your helpful comments, we improve our work from the empirical and theoretical perspectives. From the empirical perspective, we do experiments on the real-world CelebA 256 dataset and show that our drifted VESDE is a plug-and-play method without training. More specifically, the images generated by our drifted VESDE are more detailed than those of the pure VESDE baseline (shown in the PDF of the global rebuttal). From the theoretical perspective, we show that our drifted VESDE covers common diffusion models (including VP and VESDE) and goes beyond. More details and discussions are shown in the rebuttal part. We will add the above discussion to our next version and are more than happy to answer any further questions.

Thank you for your valuable comments and suggestions. We provide our response to each question below.

W1: The real-world experiments on CelebA 256.

We do experiments on the CelebA 256 dataset (a common face dataset) and show that our drifted VESDE can improve the results of pure VESDE without training from the quantitative and qualitative perspectives. Please see the experiment detail, discussion, and generated images in the global rebuttal part.

W2: The discussion on the diameter.

Since there is no existing work to discuss the lower bound of the diameter of VE-based models, we start from the VP-based models and discuss the improvement space of VE-based models to achieve the same order results compared with VP-based models.

For the reverse SDE, VP-based models achieve an optimal $d$ dependence by using the stochastic localization technique and exponential-decay stepsize [1]. As a first step, we use a uniform stepsize for VE-based models, which leads to a slightly worse dependence on $R$ and $d$. It is an interesting future work to use a more refined, time-dependent stepsize to achieve improved results for VESDE and show that VESDE performs better than VPSDE from the theoretical perspective.

For the reverse PFODE, since the Wasserstein distance can not use the data processing inequality, Thm. 6 has an exponential dependence on $R$. As discussed at the end of Sec. 6, it is possible to introduce a suitable corrector (such as the Underdamped Langevin process in [2]) to inject some small noise into the PFODE predictor, which allows the use of the data processing inequality and replaces the exponential $R$ with a polynomial one.

We will add a discussion paragraph on the diameter to make it clearer.

Q1: The guarantee under stronger metric.

(a) The pure $W_2$ guarantee.

Since we assume the manifold hypothesis, we first show how to obtain a pure $W_2$ guarantee at each setting. For the reverse SDE setting, similar to Corollary 5 of [3], by using the projection technique, we can achieve pure $W_2$ guarantee $\tilde{O}(1/\epsilon\_{W\_2}^{12})$, which has the same order with [3]. We note that the slightly worse $\epsilon$ dependence of our work and [3] is due to the relationship between $W_2$ and $\mathrm{TV}$ [4]:

For the reverse PFODE setting, similar to the tangent-based method for VPSDE [5], our results can extend to $W_p$ for any $p\ge 1$.

(b) The $\mathrm{KL}+W_2$ guarantee.

When considering reverse SDE setting, similar to [6], we can use the chain rule of $\mathrm{KL}$ divergence instead of the triangle inequality to obtain a $\mathrm{KL}+W_2$ guarantee $\tilde{O}(1/\epsilon_{\mathrm{KL}}^2\epsilon\_{W\_2}^8)$.

We will discuss the guarantee under stronger metrics in detail.

Q2 and Minor Question: the presentation.

Thanks for your helpful comments on the presentation. For our tangent-based unified framework part, we will simplify the formula of Thm. 6.1 and Coro. 6.2 and highlight the technique novelty. For the typos, we will polish our presentation according to your comments.

[1] Benton, J., De Bortoli, V., Doucet, A., & Deligiannidis, G. (2023). Linear convergence bounds for diffusion models via stochastic localization. arXiv preprint arXiv:2308.03686.

[2] Chen, S., Chewi, S., Lee, H., Li, Y., Lu, J., & Salim, A. (2024). The probability flow ode is provably fast. Advances in Neural Information Processing Systems, 36.

[3] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2022). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215.

[4] Rolland, P. T. Y. (2022). Predicting in uncertain environments: methods for robust machine learning (No. 9118). EPFL.

[5] De Bortoli, V. (2022). Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.05314.

[6] Chen, H., Lee, H., & Lu, J. (2023, July). Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning (pp. 4735-4763). PMLR.

Thank you for your valuable comments and suggestions. We provide our response to each question below.

W1 & Q1: The different $\beta_t$ for reverse SDE and PFODE: the balance between different error terms.

(a) We first recall the reverse beginning error term when considering the unified tangent-based framework:

We note that the forward and reverse processes determine the above bound simultaneously, where the exponential terms come from the bound of the tangent process (Lem. 6.3, reverse process), and the first part comes from Thm. 4.2 (forward process).

For the reverse SDE ($\eta=1$), the exponential term becomes $\exp\left(\frac{R^2}{2\sigma_{T-t_K}^{2}}\right)$, which is independent with $\beta\_t$. Hence, choosing an aggressive $\beta_t=t^2$ (here we use $\tau =T^2$ as an example) will introduce $\frac{\bar{D}e^{-T/2}}{\tau} \exp\left(\frac{R^2}{2\sigma\_{T-t_K}^{2}}\right)$ without introducing additional terms.

For the reverse PFODE ($\eta=0$), we can also choose an aggressive $\beta_t=t^2$, which leads to a $\bar{D}e^{-T/2}/\tau$ for $\frac{\sqrt{m_T}\bar{D}}{\sigma_T}$. However, since $\eta = 0$, $\exp{\int_0^{t_K}\frac{\beta\_{T-u}}{\tau}\mathrm{d}u}$ term will introduce an additional $e^{\frac{T}{6}}$ to the bound of the tangent process. We note that the key part of the convergence guarantee is to balance the discretization, reverse beginning, and early stopping error terms. Though the reverse beginning still enjoys an $e^{-T/4}\exp\left(\frac{R^2}{2\sigma\_{T-t_K}^{2}}\right)$, the aggressive $\beta_t$ will introduce an additional $e^{\frac{T}{6}}$ in the final result (since the tangent process also influences the discretization error term.). Hence, a better choice for the reverse PFODE is a conservative $\beta_t$.

(b) An interesting future work: the PFODE predictor and suitable corrector.

The above discussion shows that since the reverse beginning error is determined by the forward and reverse process at the same time, an aggressive $\beta_t$ can not be used under the PFODE setting. The complex dependency is due to the fact that data processing inequality is forbidden when considering the Wasserstein distance. As a next step, we discuss how to improve the results of Thm. 6.1 with an aggressive $\beta_t$ (the PFODE setting).

We first recall the data processing inequality: Consider a channel that produces $Y$ given $X$ based on the law $P_{Y \mid X}$. Let $P_Y$ be the distribution of $Y$ when $X$ is generated by $P_X$ and $Q_Y$ be the distribution of $Y$ when $X$ is generated by $Q_X$. Then we know that for any $f$-divergence $D_f(\cdot \| \cdot)$, $D_f\left(P_Y \| Q_Y\right) \leq D_f\left(P_X \| Q_X\right)$.

When choosing $f(x)=\frac{1}{2}|x-1|$, the $f$-divergence is TV distance. Hence, by viewing $Q_{t_K}$ as the channel, the inequality $\operatorname{TV}\left(Q\_{t_K}^{q\_{\infty}^\tau}, Q\_{t_K}^{q_T^\tau}\right)\leq \operatorname{TV}\left(q\_T^\tau, q\_{\infty}^\tau\right)$ holds, which indicates that for the $\mathrm{TV}$ distance, the influence of reverse process can be ignored when considering the reverse beginning error term. However, the data processing inequality does not hold for Wasserstein distance. To overcome this problem, an interesting future work is to introduce a suitable corrector (such as the Underdamped Langevin process in [1]) to inject some small noise into the PFODE predictor, which allows the use of the data processing inequality and achieve a polynomial sample complexity.

We will add a discussion to make it clearer.

W2: The real-world experiments on CelebA 256.

We do experiments on the CelebA 256 dataset (a common face dataset) and show that our drifted VESDE can improve the results of pure VESDE without training from the quantitative and qualitative perspectives. Please see the experiment detail, discussion, and generated images in the global rebuttal part.

Q2: The dependence on $d$ and $R$.

We recall that the result shown in [2] is $\tilde{O}\left(\frac{d^3 R^4(R \vee \sqrt{d})^4}{\varepsilon_{\mathrm{TV}}^2 \varepsilon_{W_2}^8}\right)$ and our results is $\tilde{O}\left(\frac{dR^4(d+R\sqrt{d})^4}{\epsilon\_{W_2}^{8}\epsilon_{\text{TV}}^2}\right)$. Since the image datasets are usually normalized into $[-1,1]^d$, $R\leq \sqrt{d}$. Hence, our results have the same order as [2]. We will add a discussion part about the dependence on $d$ and $R$.

Q3: The superiority of VE-based models over VP-based models.

When considering the reverse PFODE, we show the superiority of VE-based models over VP-based models. More specifically, Lem. 6.3 contains $\exp{(\int_{0}^{t_K}\beta\_{T-u}/\tau du)}$ for the reverse PFODE. For VPSDE ($\beta_t =1$ and $\tau =1$), there is an additional $\exp{(T)}$. On the contrary, VE-based models make use of the variance exploding property of VESDE, avoid this term (for example, the above term is a constant for $\beta\_t=t$ and $\tau = T^2$), and achieve polynomial $T$ dependence in final results. We note that our results also hold for VESDE ($\sigma\_t^2= t^2$), the SOTA models proposed by [3]. We will add a discussion part in the next version.

[1] Chen, S., Chewi, S., Lee, H., Li, Y., Lu, J., & Salim, A. (2024). The probability flow ode is provably fast. Advances in Neural Information Processing Systems, 36.

[2] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2022). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215.

[3] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577.

Thanks for your valuable comments and suggestions. We provide our response to each question below.

W1: The real-world experiments.

We do experiments on the CelebA 256 and show that our drifted VESDE improves pure VESDE without training from the quantitative and qualitative perspectives. Please see the experiment details in the global rebuttal.

W2: The link between drifted VESDE and VESDE and universe error analysis.

(i) For reverse PFODE, our analysis holds for pure VESDE with $\sigma_t^2=t$ or $t^2$, which covers the SOTA VESDE.

(ii) For reverse SDE, we extend our general drifted VESDE formula and provide a universe complexity for general $\tau \in [1,+\infty)$ and $\beta_t\in [1,t^2]$.

where $m_t=\exp \left(-\int_0^t \beta_s/\tau \mathrm{~d} s\right)$ and $\sigma_t^2=\tau\left(1-m_t^2\right)$.

(a) The general formula covers current models (including VE and VPSDE).

When $\beta_t=1$ and $\tau=1$, the drifted VESDE becomes VP and $m_t=\exp{(-T)}$, which leads to a logarithmic $T$ and achieve $\tilde{O}(1/\epsilon\_{W_2}^8\epsilon\_{TV}^2)$ (the same with [1]). When $\beta_t=1$ and $\tau =T$, our formula is similar but slightly better (Fig. 2 and real-world experiments) to pure VESDE (\sigma_t^2= t) and achieves $1/\epsilon\_{W_2}^8\epsilon\_{TV}^6$ results ([2] achieve $1/\epsilon\_{W_2}^8\epsilon\_{TV}^4$ since they assume strong LSI holds). For $\beta_t=t$ and $\tau =T^2$, the formula is similar to SOTA pure VESDE ($\sigma_t^2=t^2$) and achieves the first polynomial results $1/\epsilon\_{W_2}^8\epsilon\_{TV}^6$ under the manifold hypothesis. We also note that the above results holds for pure VESDE with $\sigma_t^2= t$ and $t^2$.

(b) Go beyond: When given a $\beta_t$, the optimal $\tau$ has the same order with $\beta_T$ for reverse SDE.

The key part of analysis is balancing the reverse beginning and discretization error (please see approximated score in Q2). As in (a), pure VESDE has a worse $\epsilon_{TV}$ than VP, which comes from the large reverse beginning term. For example, if $\beta_t=t$ and $\tau =T^2$, $m\_T= e^{-1/2}$ and $\sigma\_T^2=(1-e^{-1})T$, which leads a polynomial $T$ and heavily influence the discretization term. Hence, the optimal choice of $\tau=T$ instead of $T^2$ and $m\_T=e^{-T}$. Then, we achieve the same guarantee with VPSDE [1]. Similarly, the optimal $\tau$ is $T^2$ for $\beta_t=t^2$ (Thm. 5.2).

W3: The improved results for [3].

[3] considers VPSDE with the reverse SDE and achieves a guarantee with an exponential term $\exp{(1/\delta)}$. [1] achieve a pure $W_2$ guarantee $1/\epsilon_{W_2}^{12}$ by using a projection technique=, which is a direct improvement of [2]. Our general formula can also recover this result (W1), although the main point of our work is not VPSDE.

Q1: Discussion on Karras et al.

This work unifies the reverse PFODE of VP and VESDE (Eq. 4 of their work) and proves that the ODE solution trajectory of VESDE ($\sigma_t^2=t^2$) is linear and directly towards the data manifold (Fig. 3 of their work). On the contrary, the trajectories of VP and VESDE ($\sigma_t^2=t$) are not linear in most regions, which makes the denoise process difficult.

Q2: The error terms.

The reverse beginning, discretization, and approximated score error are both important. Since the sampling and learning process are relatively independent, current works usually decouple these parts. For the sample process, previous works assume an $L_2$ accuracy score [1] [3]. For the learning process, some works analyze how to use a NN to learn score [4]. We will add a discussion about error terms.

Q3: The discussion of Conforti et al.

This work considers VPSDE with reverse SDE under the finite fisher information assumption. Though this work relies heavily on the property of OU process, it is an interesting future work to analyze whether the connection in Q5 can be used to improve the results of VESDE. We also note that our work analyzes a broader area (reverse SDE and PFODE). We will add a discussion.

Q4: The novelty of our tangent-based method.

The technique novelty is our tangent-based lemma. For the PFODE, Lem. 6.3 contains $\exp{(\int_{0}^{t_K}\beta_{T-u}/\tau du)}$. For VPSDE ($\beta_t =1$ and $\tau =1$), there is an additional $\exp{(T)}$, which indicates that the previous lemma can't deal with reverse PFODE even under the VPSDE setting. To avoid this term, we use the variance exploding property of VESDE (for example, the above term is a constant for $\beta_t=t$ and $\tau = T^2$) and achieve polynomial $T$ in final results.

Q5: The connection between VP and VESDE.

As shown in Montanari, VE and VPSDE are equivalent to a change of time, which indicates the discretization analysis of these models is similar under the reverse SDE. Our universal analysis (W2) also reflects this phenomenon. However, the other key point is the balance of the first two terms, and pure VESDE performs badly. Hence, we further propose the general drifted VESDE formula, prove the optimal choice of $\tau$ faces a $\beta_t$, and provide better results. We will make the discussion of W1 and Q5 clearer.

Limitation.

Thanks for the comments on Limitation. We will discuss the benefit of our drifted VESDE formula (including VPSDE and pure VESDE) and its potential to achieve the SOTA performance.

[1] Chen et al. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. ICLR 2023.

[2] Lee et al. Convergence for score-based generative modeling with polynomial complexity. NeurIPS 2022.

[3] De Bortoli, V. Convergence of denoising diffusion models under the manifold hypothesis. TMLR.

[4] Chen et al.. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. ICML 2023.