Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers

We thank the reviewer for providing the highly inspiring feedback. In the revised paper, we have made changes based on your review comments and have highlighted all revisions in blue fonts.

Q1: Assumption 2 requires an upper bound on the score estimation error $\epsilon = \tilde{O}(T^{-2})$, which could be restrictive compared to the previous literature (e.g., Li et al. 2024b) which applies to any $\epsilon$.

A1: We clarify that for score-mismatched scenarios (which our paper considers), the score estimation error needs to be achieved at higher accuracy than the score-matched DDPM (e.g., [1]), because an extra term proportional to $\epsilon$ arises when there is score mismatch (see Lemma 2). Such a higher level of estimation accuracy also occurs in theoretical studies for accelerated DDPM samplers (e.g., [1, Theorem 3.2]). We also made the clarification in our revised draft.

Q2: The in-line equations in Section 2.1 and Section 5 are hard to follow. I suggest the authors re-organize the equations for better readability, especially by highlighting the definitions and differences of $q$, $p$, and $\hat{p}$.

A2: We have clarified the difference of notations in Section 2.2 and before Theorem 1. In particular, $Q$ is the distribution used for training the score, $\tilde{Q}$ is the target distribution, and $P$ and $\widehat{P}$ represent the sampling distribution with and without estimation error, respectively. We have also reduced the number of inline equations in Section E (prev. Section 5).

Q3: The big-O notation in this work is confusing. To name a few, in Assumptions 3 and 4, should $(1-\alpha_t)^m \mathbb{E}[\cdot] = O((1-\alpha_t)^m)$ be equivalent to $\mathbb{E}[\cdot] = O(1)$? Or do the hidden constants in these assumptions implicitly depend on $(1-\alpha_t)^m$?

A3: Thank you for your suggestion. We have made our Assumptions 3 and 4 more clear by removing the factors. In particular, Assumptions 3 and 4 can still be satisfied under two cases: (i) where $Q_0$ has bounded support for any $H$, with a special $\alpha_t$ in Equation (8) (see the proof of Theorem 4); or (ii) where $Q_0$ is Gaussian mixture and $H = [I_p, 0]$ (see Lemma 8). We have made more clarifications after stating the assumptions. Please check out the revised draft.

Q4: In Definition 1, the noise schedule needs to satisfy $\bar{\alpha}_T = o(1/T)$ which is defined as an asymptotic bound $\lim \sup | \bar{\alpha}_T / (1/T) | = 0$, while Theorem 1 presents a non-asymptotic analysis. How does the asymptotic assumption apply to the non-asymptotic analysis?

A4: Such an analysis trick has been used widely in prior literature (say, [3]); namely “non-asymptotic analysis” is applied to the regime where $T$ is large enough but finite. When $T$ is large enough, $\bar{\alpha}_T = o(1/T)$ implies that for all constant $\epsilon<\infty$ we have $\bar{\alpha}_T < \epsilon/T$. Thus, this guarantees that the initialization error does not contribute to the rate-determining terms in Theorem 1 (see Line 1057).

Q5: I suggest the authors clarify the notations or even explicitly write the constants' dependency on the hyperparameters if possible.

A5: Thank you for your suggestions! We have made our Assumptions 3 and 4 more clear by removing the factors. Please check out our revised draft. Also note that in Theorem 1, the $\lesssim$ only omits terms that decay faster than the rate-determining terms being displayed; there are no other hyperparameter-related coefficients/constants hidden in these rate-determining terms themselves.

Q6: Does $1-\alpha_1 = \delta$ in equation (8) contradict with $1-\alpha_1 \lesssim \log T / T$ in Definition 1? What can we obtain from Theorem 1 if $\alpha_t$ is chosen as (8)?

A6: Thank you for pointing this out! We agree that the $\alpha_t$ in Equation (8) does not satisfy $1-\alpha_1 \lesssim \log T / T$ at time $t=1$. However, Corollary 1 only requires such $\alpha_t$ where Definition 1 is satisfied at $t \geq 2$. We have clarified the requirement on $\alpha_t$ in Corollary 1 in the appendix.

We sincerely thank the reviewer once again for your highly inspiring comments. We hope our responses have effectively addressed your concerns. If so, we kindly ask if you might consider raising your score. Of course, we would be happy to address any additional questions or comments you may have.

Q3: I am slightly confused by theorem 2 following from theorem 1 (more specifically, Corollary 1 following from theorem 1). Theorem 2 bounds the KL between $Q_1$ (the marginal distribution of the zero-shot sampler at time 1) and $P_1$ (e.g. the time 1 marginal of the VP-SDE at time 1 where $P_0$ is the conditional distribution). Typically for all practical purposes in diffusion models $Q_1$ is simply a Gaussian and this bound can be obtained from exploring the ergodicity of the OU-process and prior work on the convergence of diffusion models. So two questions:

Question 1: Why is this bound for a general $Q_1$ relevant? We typically fix/choose $Q_1$ , so I’m a bit confused about what specific aspect of the zero-shot diffusion models considered in this work induces/requires a particular $Q_1$ which needs to be analysed.

Question 2: How does corollary 1 follow from theorem 2, do you simply just perform the same analysis in reverse time? (e.g. flip the time indices) more detail here would be useful

A3: We first clarify our notations, which largely follows from [5]. In particular, $Q$ is the distribution used for training the score, $\tilde{Q}$ is the target distribution, and $P$ and $\widehat{P}$ represent the sampling distribution with and without estimation error, respectively. In the forward process, $Q_t$ is the Gaussian perturbed version of the training data distribution $Q_0$, and $Q_T$ is very close to a standard Gaussian. In the reverse sampling process, we start from $x_T \sim \widehat{P}_T = \mathcal{N}(0,I_d)$ and goes backward in time $t$ to sequentially obtain $x_T,\dots,x_1$. Also note that $t$ is the index on the number of steps, which is different from the SDE diffusion time.

Regarding Question 1: Since we go backward in time in the reverse process, we fix/choose $\widehat{P}_T = \mathcal{N}(0,I_d)$ and provide a convergence upper bound for $\mathrm{KL}(\tilde{Q}_0||\widehat{P}_0)$ in Theorem 1. Here Theorem 1 characterizes how close the final output distribution $\widehat{P}_0$ is to the target distribution $\tilde{Q}_0$. However, for general target distributions, $\mathrm{KL}(\tilde{Q}_0||\widehat{P}_0)$ might be undefined for non-smooth $\tilde{Q}_0$ (e.g., images). Thus, in Corollary 1 and Theorem 2, we provide an upper bound using the early-stopped version $\widehat{P}_1$. In words, Corollary 1 and Theorem 2 say that the early-stopped output distribution ($\widehat{P}_1$) will be close to some distribution ($\tilde{Q}_1$), which itself is a slight perturbation of the true target distribution ($\tilde{Q}_0$). Note that such KL-Wasserstein guarantees are common for early-stopped procedures in previous theoretical investigations (e.g., [7]).

Regarding Question 2: First, our Corollary 1 follows from Theorem 1 because the Gaussian perturbed distribution $\tilde{Q}_1$ both exists and is analytic. Then, Theorem 2, whose proof is based on Corollary 1, shows the explicit dimensional dependency with a particular choice of noise schedule in Equation (8). We will provide more details in our revised draft.

Q4: How realistic is assumption 4 for zero-shot models such as Chung 2022a-b, Chung 2023 where the score mismatch can’t really be controlled as it makes rather unrealistic Gaussian approximations of the transition density in the backwards process. Assumption 4 is a rather specific bound on the mismatch; it seems its form is required to make everything else work so discussion on its relevance outside of a Gaussian setting for the prior (data distribution) would be extremely helpful.

A4: Please see our answer in A2.

We deeply appreciate the reviewer’s thoughtful and inspiring feedback. We hope our responses have satisfactorily addressed your concerns. If so, we kindly ask you to consider raising your score. Please don’t hesitate to reach out with any additional questions or comments—we would be delighted to address them.

We thank the reviewer for providing the highly inspiring feedback. In the revised paper, we have made changes based on your review comments and have highlighted all revisions in blue fonts.

Q1: The reviewer found the presentation of the paper a bit confusing. Calling methodologies such as reconstruction guidance (Chung 2023 in the paper) zero-shot (whilst potentially accurate) can be confusing and misleading for the following reasons:

• Whilst many of these methodologies do not require additional supervision (e.g. observing x,y pairs such as in CFG or classifier guidance) they do often still approximate the same quantity e.g. the h-transform / guidance term [1], ultimately with the goal of approximating the posterior score.

• Many of these zero shot approximations induce costs that are comparable to training when one considers (see wallclock times in [1]) , furthermore with methods like DPS / reconstruction guidance we can see that they are effectively fine-tuning/minimising a reconstruction loss across timesteps, whilst there's no extra data (thus arguably zero shot) , there is certainly a significant extra cost arising from this "online-finetuning".

In short, I think the paper would improve its reach/impact if it revised its presentation of what it considers to be zero-shot, or make it much more precise from earlier in the manuscript.

A1: We greatly appreciate your detailed discussion of the pros and cons of different conditional guidance methods. We employ the name “zero-shot” to highlight the fact that the approximated guidance term in our paper does not require extra training based on conditional information, which is different from other popular methods such as classifier guidance and classifier-free guidance (CFG). This terminology also aligns with some previous surveys in image editing (see [2, Section 7.1.2] and [3, Section V.E]). We have clarified our definition for “zero-shot” in our revised draft with the remark that some zero-shot methods might have additional computational costs during sampling.

While it is true that methods like DPS require “online-finetuning” that increases computation cost, there are some other zero-shot methods, including the DDNM [4] which is of primary focus in this paper, that take advantage of the linear conditional model and thus have similar computation complexity as the vanilla (unconditional) DDPM [5] (see [4, Alg. 1]). We agree that a theoretical convergence analysis of the samplers using finetuning as in [1] is an interesting future direction.

Q2: Assumption 4 seems quite strong and rather non-practical for some of these zero-shot methodologies, which make Gaussian approximations on the reverse process and thus do not induce a mismatch that can be controlled.

A2: We respectfully disagree with the reviewer’s claim that Assumption 4 is strong. In our paper, Assumption 4 has been established in two cases of zero-shot conditional sampling: (1) where $Q_0$ has bounded support for any $H$, using a special $\alpha_t$ in Equation (8) (see the proof of Theorem 4); or (2) where $Q_0$ is Gaussian mixture and $H = [I_p, 0]$ (see Lemma 8). Here the first case has wide applicability in practice (e.g., images [2-5]) and is commonly assumed in many theoretical investigations (e.g., [6]). We have made more clarifications in the revised draft after stating the assumptions.

The authors have clarified my concerns and initial misunderstanding about their posterior score decomposition (and approximation) and made the paper more accessible, so I have updated my score.

I have also re-calibrated my confidence to account for my initial misunderstanding of their work. While I believe I have a better understanding now, the initial confusion impacted my first review, and thus, I should have assigned lower confidence.

I would like to thank the authors for their patience and careful explanation of their work during the rebuttal process.

We thank the reviewer for providing the highly inspiring feedback. In the revised paper, we have made changes based on your review comments and have highlighted all revisions in blue fonts.

Q1: I am a bit afraid that this paper overclaims the problem settings it actually solved. Specifically, this paper claims to consider a general score-mismatched setting. Although it provides a convergence analysis (Theorem 1) for such a setting, the assumption, e.g., (Assumption 4) required, is highly artificial. Besides, it can hardly find an algorithm to achieve Theorem 1, since authors do not provide any approximation for $\nabla \ln \tilde{q}_t$ under a general score-mismatched setting ($y=h(x_0)$). Instead, I think this paper more focused on the linear setting, i.e., $y = Hx_0 + n$ (Eq.(4)).

A1: We respectfully disagree with the reviewer. (i) Our score-mismatched analysis is indeed applicable to general conditional models, including non-linear ones, as long as there is some mismatch in the scores. In particular, Assumption 4 naturally captures to what extend the mismatch is, which needs to be bounded (in the $L^\ell(\tilde{Q}_t)$ space) so that accumulating $t=1,\dots,T$ the bias will not diverge. Note that Assumption 4 does not assume a linear conditional model. (ii) We acknowledge that it is hard to find an algorithm to achieve Theorem 1 for the general score-mismatched setting ($y=h(x_0)$). But our paper did not overclaim that we designed such an algorithm.

We re-emphasize that our contribution lies in convergence analysis, starting from a general scenario and specializing to the linear setting. And further, we provide a bias-optimal sampler design for the linear case. We believe what we claim in the paper matches the results we develop.

Q2: I am curious about whether the problem setting is really useful beyond the super-resolution problem. Since the conditioning (y) is usually correlated with the random variable (x) implicitly. I suggest authors provide more examples of utilizing zero-shot samplers to solve score-mismatch diffusions.

A2: The linear model is useful for a number of computer vision applications, including image super-resolution, image inpainting (where $H$ is 0-1 diagonal in both cases), image colorization (where $H$ is a pixel-wise operator with $[1/3, 1/3, 1/3]$), image deblurring (where $H$ is the blurring filter) (see [1, Section D] and [2, Section 3.2]). The linear model is also useful for medical image reconstruction [3], where $H$ corresponds to the phases from Fourier Transform. The linear model is also the main focus in many empirical zero-shot diffusion samplers, such as DDRM [1], DDNM [2], MCG [4], and $\Pi$GDM [5]. These facts inspire us to focus primarily on linear conditional models.

Q3: The establishment of Assumption 4 requires strict assumptions. It not only constrains the unconditional distribution $Q_0$ but also constrains the $H = [I_p, 0]$. I hope the authors can demonstrate whether the results can extend to a general $H$ and how.

A3: Assumption 4 is not restrictive but rather soft. In our paper, Assumption 4 has be established in two cases: (1) where $Q_0$ has bounded support with a special $\alpha_t$ in Equation (8) (see the proof of Theorem 4); or (2) where $Q_0$ is Gaussian mixture and $H = [I_p, 0]$ (see Lemma 8). In the first case, we do not require any particular form for $H$. Also here, the assumption that $Q_0$ has bounded support has wide applicability in practice (e.g., images [1-5]) and is commonly assumed in many theoretical investigations (e.g., [6]). For the second case, the proof of Lemma 8 can be easily extended for general $H$ (since $||H^\dagger H|| = ||I_d - H^\dagger H|| = 1$). We have added this clarification in the revised draft.

Q4: The final KL divergence bound between ground truth and generated samples is unsatisfactory. Although it is an upper bound, it contains a factor $O(d)$. Such a result will make me concern whether a zero-shot sampler is really effective for mismatch-score settings.

A4: On the theory side, linear dimensional dependency is established in many state-of-the-art analyses of score-matched DDPMs (say, [7,8]). While there are some works to further shrink $d$ dependency by assuming an underlying low-dimensional structure, the current factor of $O(d)$ is considered to be satisfactory when such a low-dimensional structure is not assumed. Also, in empirical studies, the performance of zero-shot samplers, in particular that of DDNM [2], has been shown to be very effective. Along this line of research, our BO-DDNM further reduces the bias of DDNM both theoretically and empirically.

We thank the reviewer for providing the highly inspiring feedback. In the revised paper, we have made changes based on your review comments and have highlighted all revisions in blue fonts.

Q1: Assumption 2 is not realistic: Assumption 2 measures how much the approximated conditional expectation deviates from the true $\mu_t$. The expectation in assumption 2 is taken with respect to $\tilde{Q}$ the target distribution, which is unknown during training. Assumption 2 can thus not be guaranteed to hold.

A1: In learning theory, all performance guarantees are obtained by assuming that the ground-truth scenarios satisfy the claimed assumptions. This does not require that those assumptions can be checked during training, as long as the execution of the algorithm does not need such information. Thus, in almost all literature on theoretical studies of DDPMs, the assumption involves the target distribution, which is unknown during training.

In practice, Assumption 2 is very likely to hold as long as $\tilde{Q}$ is close to $Q$. Also note that in the past, as a first attempt to investigate score-matched DDPMs, [1] assumes that the score function is $L^\infty$-accurate. This assumption is sufficient for our Assumption 2 to hold. In particular, in case of zero-shot conditional sampling, we only consider those $y$’s such that Assumption 2 (and 5) holds.

Q2: Moreover, the authors state mistakenly, as far as I understand, that Assumption 5 is the same as Assumption 4 when taken under different distributions. Assumption 5, under $Q_t$, is a classic assumption and, as it relates closely to the training loss, is a realistic assumption. Assumption 4 on the other hand needs more justification.

A2: We did not find the suggested claim in our paper that “Assumption 5 is the same as Assumption 4”. Instead, we claim that “Assumption 5 directly implies Assumption 2,” which is proven (straightforwardly) in Equation 9.

Assumption 4 can been verified in two cases of zero-shot conditional sampling: (1) where $Q_0$ has bounded support with a special $\alpha_t$ in Equation (8) (see the proof of Theorem 4); or (2) where $Q_0$ is Gaussian mixture and $H = [I_p, 0]$ (see Lemma 8). Here the first case has wide applicability in practice (e.g., images [2]) and is commonly assumed in many theoretical investigations (e.g., [3]). We have added this piece of clarification in the revised draft.

Q3: A minor modification of prior work with assumptions to control the additional terms: Theorem 1 is obtained through modification Liang et al 2024 by assuming the target distribution is different from the training one. The simple additional terms that appear due to the modification are controlled using a series of assumptions. This limits the technical novelty over Liang et al 2024.

A3: We respectfully disagree with the reviewer. (i) Theorem 1 identifies the bias terms due to score mismatch via a non-trivial analysis. As we outlined in Section E (prev. Section 5), each of the four steps in the proof of Theorem 1 requires us to establish non-trivial bounds due to mismatched scores. For example, the mismatched scores require us to provide an upper bound for the expected tilting-factor difference in terms of two different conditionals: $P$ and $\tilde{P}$. In comparison, we directly have $P = \tilde{P}$ for score-matched scenarios. Because of this difference, different from the score-matched case where we have centralized moments, here we need to upper-bound all higher-order non-centralized moments, for which we employ non-trivial combinatorial techniques (Lemma 4). (ii) Those additional bias terms, which do not exist for the score-matched models analyzed in [4], are not simple, and analyzing them does not come readily from the assumptions. In general, the bias in Theorem 1 may not have a constant upper bound. We show the existence of such a constant upper bound for two noise schedules: the constant noise schedule (Lemma 10) and that in Equation (8) (Lemma 7). Here the latter schedule corresponds to the commonly used exponential-decay-then-constant noise schedule (e.g., [5]). (iii) We also note that the novelty of our paper goes beyond Theorem 1. For example, for the zero-shot conditional sampling problem, we provide a novel design of a zero-shot sampler, as long as explicit parameter dependencies of the sampler (see Section 1.1). Both of these contributions are not captured by Theorem 1 nor by [4].

Dear Reviewer nM2R,

We've taken your initial feedback into careful consideration in our response. Could you please check whether our responses have properly addressed your concerns? If so, could you please kindly consider increasing your initial score accordingly? Certainly, we are more than happy to answer your further questions.

Thank you for your time and effort in reviewing our work!

We thank the reviewer for providing the highly inspiring feedback. In the revised paper, we have made changes based on your review comments and have highlighted all revisions in blue fonts.

Q1: Smoothness Requirements: The assumption requires the existence and control of higher-order derivatives of the log densities. Distributions that are not sufficiently smooth (e.g., those with discontinuities or singularities) may not satisfy this assumption.

A1: Note that Assumption 3 is not restrictive. When $\tilde{Q}_0$ is non-smooth, we can show that Assumption 3 holds as long as $\tilde{Q}_0$ has finite variance (not only bounded support, see Lines 1219-1229), for a particular choice of $\alpha_t$ in Equation (8). Note that this $\alpha_t$ corresponds to the early-stopped procedures in the literature, which is a common practice for non-smooth target distributions [1-2].

Q2: Scaling with $(1-\alpha_t)^m$: When $\alpha_t$ approaches 1, the factor $(1-\alpha_t)^m$ becomes very small. If the expectations do not decrease at the same rate, the assumption may not hold. This can occur in practice when $\alpha_t$ is close to 1, as in the noise schedule Equation (8) in the paper.

A2: Note that this scaling factor appears on both sides of the equations in Assumption 3. Thus, a sufficient condition for Assumption 3 is that the two expected partial derivatives are bounded in $L^\ell$ space. Indeed, with the noise schedule in (8), we can verify Assumption 3 for any $\tilde{Q}_0$ with finite variance.

Q3: High Dimensionality Issues: In high-dimensional settings (large $d$), the behavior of the derivatives can be more complex. The curse of dimensionality may make the assumption unrealistic because the number of derivatives grows exponentially with $d$.

A3: In [3, Lemmas 15 and 17], an upper bound is shown for all expected values of higher-order derivatives with only polynomial dependency on $d$ (rather than “exponential” as suggested). Such polynomial dependency is due to the nice Gaussianity of the perturbation in the forward process.

Q4: Applicability to other distributions: The assumption may not be valid for distributions outside certain classes, such as those with multimodal densities where modes are not well-separated, or distributions with skewness and kurtosis that affect the derivatives.

A4: Assumption 3 can be satisfied for any $\tilde{Q}_0$ has finite variance with the $\alpha_t$ in Equation (8) (see Lines 1228-1238). Here even though $\log q_0$ might not be smooth, all Gaussian perturbed $\log q_t$’s are still infinitely smooth. Such $\tilde{Q}_0$ constitutes a large class of distributions, including multimodal densities and those with non-smooth derivatives.

Q5: Avoiding overstatements of contributions: In Section 1.1, the paper claims to provide convergence bounds for general target distributions with finite second moments (e.g., Line 92-93). However, their analysis requires regular high-order derivatives for the score functions, which significantly restricts their results to smooth distributions. Adjusting the claim accordingly would provide a more accurate representation of the scope of the contributions.

A5: See our response in A4. While Assumption 3 is necessary for the proof of Theorem 1, this assumption can be satisfied for a fairly large class of distributions.

Q6: Insufficient Discussion on the obtained bounds: The paper lacks sufficient discussions on the practical implications of the bounds. For example, how do the obtained bounds compare to conventional conditional sampling methods, such as Langevin dynamics? How can the bounds explain the success of conditional diffusion models over other sampling methods?

A6: First, to the best of our knowledge, there are no theoretical bounds for conditional samplers using Langevin dynamics to compare with, in particular for the inverse problem. Also, to be clear, the goal of this paper is not to compare diffusion-model based conditional samplers with other conditional samplers, say those using Langevin dynamics. The goal, instead, is to investigate the performance of zero-shot conditional diffusion samplers when there is score-mismatch, and to compare zero-shot samplers with those diffusion samplers knowing full conditional scores. With our result in Theorem 1, we show that, different from these samplers knowing full scores, zero-shot samplers introduce an asymptotic bias which is a function of the score-mismatch. Then, we further minimize the bias with the proposed BO-DDNM sampler.

Dear Reviewer w6Vp,

We've taken your initial feedback into careful consideration in our response. Could you please check whether our responses have properly addressed your concerns? If so, could you please kindly consider increasing your initial score accordingly? Certainly, we are more than happy to answer your further questions.

Thank you for your time and effort in reviewing our work!