Solving Inverse Problems via Diffusion Optimal Control

We highlight our gratitude that the reviewer appreciates both the theoretical and empirical results of our work, and for bringing several insightful shortcomings of our work to our attention. Below we respond to the reviewers concerns in a point-by-point basis.

Mathematical formulations are complex and potentially challenging to implement. We agree that the mathematical framework behind optimal control theory is sophisticated, and differs from that of standard diffusion models, which comes from nonequilibrium thermodynamics. However, our approach has several distinct advantages. First, we can treat the generative process as a black box dynamical system, which vastly abstracts the pre-existing mathematics in the reverse diffusion process. Practitioners may thus choose between standard diffusion modeling and optimal control-based modeling based on their expertise. Second, we can leverage extensive research from the optimal control community to improve diffusion-based inverse solvers. Third, we are able to sidestep the intractability of the conditional score function, as discussed in Section 4.

Computing Jacobian and Hessian matrices and their inverse may have significant computational demands. We note that even a rank-one approximation of these matrices is sufficient, with further increases in rank providing quickly diminishing returns (see Table 4 in Appendix). Using modern randomized linear algebra libraries (e.g. randomized SVD), this brings the computation of the Hessian matrices to cost $\mathcal{O}(d)$, where $d$ is the data dimension. Therefore, the cost of our algorithm is equivalent to the cost of, e.g., performing a DPS step. (For more details, see a thorough complexity analysis in Appendix D.2.)

Including more diverse datasets would provide a more comprehensive evaluation. Thank you for this insight. We agree, and additionally include results on ImageNet, and more nonlinear settings. Please see the main rebuttal and PDF.

Purpose and mechanism of injected control vectors in the reverse diffusion process. The control vectors u_t are simply perturbations to the original unconditional diffusion process, and are widely used in conditional sampling with diffusion models. Analogues include the conditional score term in DPS (Chung et al., 2022), the classifier gradient in classifier guidance (Dhariwal and Nichol, 2021), or the classifier-free guidance term (Ho and Salimans, 2022).

How are $V_x$ and $V_{xx}$ used in the optimal control framework? In optimal control, $V(x)$ is the value function, which intuitively represents "desirability" of each state x in the dynamical system to the user. In this case, $x$ is the reverse diffusion iterate. $V_x$ and $V_{xx}$ are simply derivatives of this value function. We can think of $V(x)$ as the negative loss of the optimal control system. Therefore we use the derivatives $V_x$ and $V_{xx}$ much like they are used in an optimization framework to guide a solver towards (locally) optimal solutions.

Please indicate the output of Algorithm 1. The output of Algorithm 1 is the perturbed dynamics ${x’_t}_{t=1}^T$. The final iterate of which is the solution to the inverse problem. Thank you for this comment, we have clarified this point in the manuscript.

High level intuition on the adaptive optimizer for action updates. Note that the default choice of optimizer for iLQR implementations is the backtracking line search. We instead use an adaptive optimizer, and motivate this design decision with three main observations.

First, recall that given the update direction $v$ the backtracking line search finds the optimal step size $\lambda$ to update the actions by applying $u’ = u + \lambda v$ to our reverse diffusion process. Evaluating each $\lambda$ requires re-computing the inverse problem loss under the new proposed $u’$, we would have to re-run the reverse diffusion process, which is quite expensive.

Second, the backtracking line search is used with generally smooth landscapes, and not as amenable to highly nonconvex settings. Since each marginal density of the diffusion process is essentially the data distribution convolved with a Gaussian distribution, it is highly nonconvex (especially at the low noise regime) and thus ill-suited for our setting.

Third and last, since the diffusion process occurs on images which can be very high dimensional, our optimization space is also very high-dimensional. Adaptive optimizers such as Adam, Adagrad, RMSprop, etc. are specially designed for deep learning settings where there is a similarly high dimensional optimization space. Therefore, it is a natural choice for our problem setting.

We greatly appreciate the reviewer’s positive assessment of theoretical contributions, for their in-depth reading of our manuscript, and for their suggested improvements. We respond to comments in detail below.

The runtime of Algorithm 1 seems high. Without taking any approximations of the Hessian and Jacobian matrices in Eqs. 19-25, Algorithm 1 will indeed be relatively costly, compared to the other algorithms we compare against in Table 1. However, we maintain that our method is not significantly more expensive than competing methods. For details, see computational complexity analysis in Section D.3. Indeed, under the $T=50$ configuration, our method is slightly more expensive than competing methods. That being said, we run an equivalent runtime/flops budget in the Ours (T=20) case in Table 1, where we restrict our method to 1000 NFEs, which take up the majority (~97%) of the runtime of our algorithm. We still show significant gains over DPS, and comparable performance against the most recent state-of-the-art methods on FFHQ256-1K. Empirically, we find that our method has a similar runtime to DPS on an A6000 GPU (130s vs 125s).

iLQR could require a large number of iterations to converge. Indeed, iLQR could take many iterations to converge. However, we observe near convergence for nearly all of the settings we consider in our work. Moreover, true convergence is not necessarily desirable, since the forward operator measurements $y$ are noisy. Therefore, a fully converged iLQR method would overfit to the noisy, and produce inferior solutions.

...(and all nice things discussed in Section 4. rely on this convergence). We respectfully disagree. A central claim in our paper, and in Theorems 4.1-4.3 in Section 4 is the ability to compute the conditional score function $\nabla_{x_t} \log p(y | x_0)$. This claim does not rely at all on the convergence of the iLQR algorithm. In fact, this quantity is a fundamental property of the backward pass of the iLQR algorithm (the second loop in Algorithm 1). Therefore, we obtain the true $\nabla_{x_t} \log p(y | x_0)$ in every pass of our the iLQR algorithm, including the first step.

Furthermore, each iteration needs to be $\Omega(T)$ matrix solves which can be quite slow. This is true. We agree with the reviewer's statement. However, the constant swallowed by the $\Omega$ notation is important, and rather small here. After an ablation study, we found (somewhat surprisingly) that even a rank-1 approximation of the relevant matrices is sufficient to obtain competitive results on our benchmarks (Table 4). Under a rank-1 approximation, many of the matrix operations are simply $\mathcal{O}(d)$, where $d$ is the size of the image.

It would be interesting to see ablation studies on the effect of num_iter in Algorithm 1. This is a good point. We provide the ablation study on the FFHQ256-1K dataset with the super-resolution task, letting $T=50$ and $k=1$. Performance is evaluated with the LPIPS metric.

No analysis of the approximation error of iLQR (the first and second-order Taylor approximations) in the studied setting. We respectfully disagree. We do analyze the approximation quality of our proposed iLQR Algorithm 1 in Theorems 4.1 and 4.3. Here, we show that, in the deterministic setting, the solution $\mu_t$ is precisely the desired conditional score $\nabla \log p(y | x_0)$, which is the quantity desired in DPS-based solvers at each step. Therefore, in the deterministic setting, the approximation error arises entirely from the discretization error of the numerical solver for $x_0$ itself. In the stochastic setting, the approximation error will also come from the randomness of the diffusion process.

What is the rough runtime of each method in Table 1? Below we report the approximate runtimes for each method on an NVIDIA A6000 GPU. The first two are ours with different choices of $T$.

Elaboration on $p_t(x|y)$ notation. In general, $p_t(x|y)$ and $p_t(x)$ do come from a family of distributions, and refer to the marginal distribution of the conditional diffusion process at time $t$. Each $x_t \sim p_t(x|y)$ can be obtained, given $x_0$, via $x_t = \alpha x_0 + \sqrt{1 - \alpha} \epsilon$, where $x_0 \sim p(x | y)$ (the distribution of our solution set), and $\epsilon \sim \mathcal{N}(0, \mathbf{I})$.

In (29), what is the meaning of $\nabla_{x_t} \log⁡ 𝑝(𝑦|𝑥_0)$? We apologize for the confusing notation. This is the derivative of the conditional log likelihood $\log p(y | x_0)$, which depends on $x_0$, which in the reverse diffusion process depends on $x_t$. We thank you for this question and have clarified this notation in the text immediately following (29).

What is a feasible solution in Line 149? We mean feasibility in the constrained optimization sense – a solution is feasible when it satisfies all the constraints of the problem, i.e., $y \approxeq f(x)$. Since our proposed algorithm is closed loop (i.e., changes at each step $t$ depend on the final error at $t=0$), the algorithm can make global modifications to the diffusion trajectory at each $t$. To our knowledge, this is not possible for general inverse problems with competing methods like DPS, MCG, etc.

Line 176: Why is $\log⁡ 𝑝(𝑦|𝑥_0) = \log⁡ 𝑝(𝑦|𝑥_t)$ an assumption, not a consequence? Indeed, $\log p(y | x_t) = \log p(y | x_0)$ is a consequence under the deterministic ODE dynamics. However, we also consider the setting of DPS, where the process is stochastic. Under this setting, there may be more than one $y$ (and thus $x_0$) associated with a given $x_t$, since $p(x_t|x_0)$ is Gaussian and supported everywhere. In this case, the equality is no longer guaranteed, but is assumed in many prior works (e.g., DPS).

Line 204: What is the purpose this experiment with randomly initialized weights? Indeed, this application is itself not meaningful. We meant to demonstrate DPS samplers require a well-approximated score model to function properly, whereas we do not. Of course, the performance of our model also improves with a better diffusion prior, but the deterioration when the score is not well approximated is more graceful than DPS (i.e., in less represented parts of the dataset). This can be useful when using a general inverse solver (e.g. ImageNet or foundation model) and solving for an image from an unknown or poorly approximated distribution.

To further illustrate our point, we also demonstrate that the same trend occurs even when the diffusion model weights are initialized from a pretrained model, but from a different distribution than the source image. In Figure 10 in the rebuttal PDF, the right hand block shows an inverse problem setting for reconstructing ImageNet images, using an FFHQ model. As expected, our proposed method outperforms DPS on this out-of-domain inverse problem setting.

We thank the reviewer for their positive assessment of our work, and their insightful critique. Below we provide a point-by-point response to the reviewer's discussion points.

Main weakness is the computational cost of the algorithm, e.g., computing Hessians. Approximation error is not well understood. Indeed, the Hessians would be very difficult to compute in their entirety. However, we want to mention that we do investigate the computational trade-off that occurs when approximating the Hessian $V_{xx}$. Our primary method technique is the randomized low rank SVD (Halko et. al, 2011), which provides very strong approximation guarantees when there does exist low rank structure in the matrix. Indeed, the Hessians of overparameterized neural functions have been well known to be low rank (Sagun et. al, 2017), and we also verify this empirically in Table 4 in the Appendix, where we see that the algorithm performance does not deteriorate with sparser representations, in terms of rank. Moreover, even letting the rank $k = 1$ obtains very strong performance (see also Table 1). We generally found in our experiments that a low-rank approximation is sufficient to impose the quadratic trust-region optimization to our model, and does not meaningfully deteriorate performance.

Understanding the interplay between the number of diffusion steps $𝑚$ and $𝑇$. We generally find that, fixing a computational budget (e.g., 1000 NFEs), the number of diffusion steps and number of iLQR iterations can quite significantly affect the final outcome of the model. The key observation is that the iLQR objective optimizes strictly the inverse constraint, whereas the diffusion model acts as a regularizer. With a fixed budget, increasing one will come at the cost of the other. Therefore, if the measurement is very noisy, or the solution is known to be well-supported by the diffusion prior distribution, then one should take a large number of diffusion steps, and fewer number of iLQR iterations. Conversely, if the measurement is noise free, or the prior is not very informative, then the reverse should be done.

Equivalent budget analysis. Is the computational cost equivalent? We perform equivalent budget analysis in two ways. First, we note that theoretically, our Algorithm 1 costs $\mathcal{O}(nm(k^3 + kd^2))$ in terms of FLOPS and $\mathcal{O}(mnk)$ in terms of neural function evaluations (NFEs). Perhaps interestingly, we found that NFEs dominate the wall-clock time of the algorithm, accounting for 97%. We let $k=1$ in all reported experiments outside of our ablation study in Table 4. More details can be found in Appendix D.2.

Empirically, the results of our equivalent budget analysis is summarized in the Ours (T=20) entry Table 1. As discussed, neural network calls take up ~97% of the wall-clock runtime of our solver. Therefore, we measure computational budget by the number of neural function evaluations (NFEs) of the method. DPS, MCG, DDNM, and DDRM allow a computational budget of 1000 NFEs, and most other methods in Table 1 follow a similar guideline. Therefore, the entry “Ours (T=20)” runs with 50 iLQR iterations and T=20, satisfies an equivalent budget to these models. We see that under this restricted budget our method beats out several prominent methods, including DPS, while comparing favorably against other state-of-the-art methods.

Have the author really explored the hyper-parameter space for these methods. Can the authors provide some insight on how to choose the key parameters (𝑚,𝑛,𝑘)? Please see Tables 4, 5, and 6 for an in-depth exploration of the main hyperparameters of our method.

We assume the reviewer is using the notation from Appendix D.2 for $(m, n, k)$. Fixing a computational budget $\mathcal{N}$, there is a clear trade-off between the number of diffusion steps $m$ and iLQR iterations $n$, since they are inversely related. More diffusion steps bias the model towards the diffusion prior, whereas more iLQR iterations bias the model towards the observation $y$.

When $m$ is small, DPS and PSLD should outperform this method with comparable cost. We in fact observe the reverse phenomenon with $m$! In other words, our model actually outperforms DPS at low $m$. We conduct the simple comparison below on the super-resolution task on FFHQ. We fix $n = 50$, and report LPIPS.

The metrics (e.g., LPIPS, PSNR, SSIM) are mentioned but not described. Thank you for pointing this out. We agree that this is can be confusing and have added references to the LPIPS, PSNR, and SSIM metrics.

We thank the reviewer for their appreciation of the optimal control perspective, and for their insightful discussion and bringing many recent works to our attention. We respond to comments in detail, on a point-by-point basis below.

The method is well-backed but might be computationally exhausting. We maintain that our method is not significantly more expensive than competing methods. For details, see computational complexity analysis in Section D.3, where we show that iteration-for-iteration, our method has similar costs to DPS. Indeed, under the $T=50$ configuration, our method will run for more iterations. Moreover, under the $T=20$ configuration in Table 1, our method has a very similar runtime as DPS (130s vs 125s on an NVIDIA A6000 GPU).

Quantitative results on different datasets and nonlinear inverse problems are lacking. Thank you for this suggestion. We have added experiments on ImageNet 256 x 256-1K and the phase retrieval and nonlinear blurring inverse problems, and find that our algorithm compares very favorably against existing methods.

Algorithm 1 takes 2500 NFEs in the $T=50$ case in Table 1. Indeed, the reviewer is correct. We see that reporting only $T$ may not show the full picture, and have updated Table 1 to show NFEs. The new Table 7 (see main rebuttal PDF) displaying ImageNet results already shows $NFEs$.

What is the computational efficiency of the proposed method? While the full matrix computations in equations 19-25 would be expensive to compute in full, we find that low rank approximations are very effective and result in minimal deterioration of the algorithm performance (for more details, see Appendix D.3 and specifically Table 4). Moreover, we find that the algorithm has the same computational complexity as most existing algorithms (e.g., DPS), and is similarly dominated by neural network calls (i.e., 97% of the wall-clock time is dominated by NFEs). Therefore, we also use NFEs to measure runtime.

I would like to see a more detailed comparison of the method with other baselines like DPS in terms of time. Under the $Ours (T=20)$ configuration in Table 1, we take the number of iterations to be $50$, resulting in $1000$ NFEs. Under this equivalent computational budget to other methods in Table 1, we show that we still outperform prominent methods such as DPS and DDRM, while comparing favorably to state-of-the-art algorithms. Finally, we verify that the computational budget is indeed equivalent by comparing our method to DPS and MCG, letting $T = 1000$. On an NVIDIA A6000 GPU, all three methods take ~120s to run.

Please discuss advantages over [1], [2], [3] and [4]. We thank the reviewer for bringing these works to our attention. After careful study, we note that [1, 3, 4] all rely on Tweedie’s formula to predict an estimated $x_0$, given $x_t$, to formulate the correction step at each time of the reverse diffusion process. Therefore, these methods all suffer from the same pitfall of DPS (and MCG, and other algorithms of the same vein), where $\nabla log p(y | x_0)$ must be approximated.

We do observe that [2] stands out from the other three in that does not require this approximation. However, the variational framework that is proposed is itself intractable in their inner optimization loop, and they require a stop-gradient on the score network to maintain tractability. Overall, these related works are important to discuss, and we have added these references to our manuscript.

I thank the authors for their hard work on the rebuttal and appreciate the additional results. However, I still have concerns about two points:

• Insufficient baselines: While I agree that DPS-like baselines share the same issues due to the use of Tweedie's formula, recent works like FPS have shown significant performance improvements. Additionally, Resample (https://arxiv.org/abs/2307.08123, ICLR 2024) is an advanced method of PSLD, as both approaches are latent-based.

• Unsatisfactory results: In Table 1 of the paper, the proposed method (T = 20) does not outperform either PSLD or DPS and is not even competitive. I wouldn't expect a significant difference between the FFHQ (Table 1 in the paper) and ImageNet datasets(in the rebuttal pdf) to change this outcome.

Given these considerations and other reviewers' comments, I retain my score.

We would like to express our gratitude that the reviewer appreciates both the theoretical and empirical results of our work, and for bringing several insightful shortcomings of our work to our attention. Below, we address the reviewers concerns on a point-by-point basis.

High computational cost:

a) Jacobian matrix computation can be costly, and low rank approximations bring additional error. Indeed, Hessian matrices in iLQR are costly at full rank. Moreover, we agree that low rank approximations generally introduce some error to the computation. However, we maintain that this error is negligible. In our ablation study, we see that even a rank-one approximation of these matrices is sufficient, with further increases in rank providing quickly diminishing returns (see Table 4 in Appendix). Using modern randomized linear algebra libraries (e.g. randomized SVD), this brings the computation of the Hessian matrices to cost $\mathcal{O}(d)$, where $d$ is the data dimension. Therefore, the cost of our algorithm is on par with competing methods, e.g. DPS. (For more details, please find a thorough complexity analysis in Appendix D.2.) We also find that there are few hyperparameters introduced by these approximations, and the sensitivity of the algorithm to these hyperparameters is low (Tables 4, 5, and 6).

b) The proposed algorithm is iterative over the simulated trajectory. Reported $T$ in Table 1 may not be directly comparable. The reviewer is correct, our Algorithm 1 requires multiple passes through the diffusion trajectory. Our intention with displaying the time steps $T$ in Table 1 is to show the stability of our algorithm even at very low time steps. This is a nice property to have, and can be useful in certain cases. For example, DPM-Solver (Lu et. al, 2022) is known to be unstable at >100 timesteps, and is usually used for T ~ 20. Methods like DPS will struggle in this regime. That being said, we do agree with the reviewer that in terms of computation time, the number of neural function evaluations (NFEs) is also important to consider, and have added it to Table 1, as well as the new Table 7 in the rebuttal PDF. Overall, we find that our model performs favorably against other methods under an equivalent computational budget (the T=20 case in Table 1 — and 1000 NFEs), and establishes a new baseline at T=50 and 2500 NFEs.

More complex dataset than FFHQ, e.g., ImageNet. We agree that further experiments can demonstrate the generalizability of our results. To this end, we evaluate our model on the ImageNet-1K dataset replicating the experimental setup in (Chung et. al, 2022), and on some nonlinear forward operators. Please see the general rebuttal and PDF.

Minor suggestion: More introduction for the optimal control part in the main paper. We fully support this suggestion. As the reviewer has noticed, we moved a large amount of the introduction to the appendix due to page constraints. However, following this suggestion we have included a slightly more intuitive discussion on iLQR and its mechanism in Section 2.3.

Only comparing diffusion timesteps is unfair. Please provide a second form of comparison for sampling time. We agree, and again note that the “Ours (T=20)” entry in Table 1 is run for 50 trajectory iterations, yielding 1000 NFEs. This results in roughly the same sampling time (e.g., on an A6000, DPS and our method both take ~120s to run). We also maintain the same 1000 NFE budget in the new Table 7 showcasing results on ImageNet.

Robustness to different initializations. We currently initialize our actions as identically 0, and our states as i.i.d. normal (as prescribed by the DDPM algorithm). In the rebuttal Figure X we show the result of the algorithm with different state and action initializations. We currently do not fix the seed of the state initializations, and our algorithm obtain similar results on each run. Therefore we find our method to be relatively robust to different initializations.

We thank the reviewer for their helpful discussion and insightful critique. We hope to clarify some of the points of the paper, and alleviate the reviewer's concerns below in a point by point response.

Only a single a single image is provided in Figs. 3 and 6. In the rebuttal PDF Figures 9 and 10, we provide further experiments that demonstrate the concepts in Figure 3 (accuracy of the predicted score) and Figure 6 (robustness to the approximated score), respectively.

No theoretical results for such robustness [to the score approximation]. This theoretical statement comes from simple fact that DPS (and related algorithms) sample from p(x|y) by solving the conditional reverse diffusion process (Eq. 1), where $\nabla \log p_t(x_t)$ is approximated by the neural score function $s_\theta$ (equivalently $\epsilon_\theta$, $v_\theta$, etc. in noise- and v- prediction models, respectively, up to a simple reparameterization). Therefore, the performance of DPS (and related algorithms) depends on the approximation $s_\theta \approx \nabla \log p_t(x_t)$. Our algorithm makes no such assumption, and is thus robust to this approximation error. We further illustrate this point with more examples in the rebuttal Figure [x].

Methods should be compared in terms of performance vs runtime / flops and no the number of diffusion steps. We do run an experiment under an equivalent runtime/flops budget in the Ours (T=20) case in Table 1, where we restrict our method to 1000 NFEs, we still show significant gains over DPS, and comparable performance against the most recent state-of-the-art methods on FFHQ256-1K. Empirically, we find that our method has a similar runtime to DPS on an A6000 GPU (130s vs 125s). That said, we do agree that listing only the number of diffusion steps can misrepresent our method. Therefore, we have added NFEs to Table 1 (just like Table 7 in the rebuttal PDF).

The proposed method is significantly more expensive than competing methods. We maintain that our method is not significantly more expensive than competing methods. For details, see computational complexity analysis in Section D.3, where we show that iteration-for-iteration, our method has similar costs to DPS. Indeed, under the $T=50$ configuration, our method will run for more iterations.

The conditional score function remains intractable in the proposed method. We respectfully disagree. Let us first follow DPS (Chung et. al, 2022) and define the conditional likelihood as $p(y | x_0)$. Therefore, the conditional score function at time t is $\nabla_{x_t} \log p(y | x_t)$. Unlike DPS, we consider both deterministic and stochastic dynamics in our theory. Under deterministic dynamics, we first note that the conditional likelihood is tractable, since $p(y | x_t) = p(y | x_0)$ — i.e., $x_0$ is determined by $x_t$, by definition — and $p(y | x_0)$ can be obtained via the forward rollout of Algorithm 1. Now, we have the distinct advantage over DPS where $\nabla_{x_t} \log p(y | x_0)$ is exactly backpropagated to $x_t$ (i.e., the backward pass of Algorithm 1, see Theorems 4.1-4.3). For this reason, we also maintain that the conditional score is tractable. This claim cannot be made by DPS, because 1) DPS is derived from stochastic dynamics and 2) DPS relies on Tweedie’s formula at each step to “approximate” the gradient of the conditional likelihood.

The statement that our model estimates $x_0$ exactly, rather than forming an approximation sounds misleading. We also respectfully disagree here. We do estimate $x_0$ exactly. Note that $x_0$ as defined in DPS (as well as our work) is $x_0 | x_t$. In other words, the denoised image given the current noisy image $x_t$ at time $t$. Observe that we do obtain this term exactly in each iLQR step, up to the discretization error of the diffusion solver. That said, we have added the qualification “up to the discretization error of the numerical solver” where applicable. We note that we do emphasize the discrete nature of our method in the abstract and throughout the paper, such as lines 67-68, 115, and 265.

With DPS one could also (uselessly) obtain an ODE estimate of $x_0$ and then compute the scores from there. Indeed, this is possible --- but like the reviewer says, uselessly slow. The impracticality of this strategy illustrates the advantage of our method. Using iLQR, we are able to combine the prediction of $x_0$ in the forward rollout with the score computations used in the feedback step in one single sweep (Algorithm 1). To see this, observe that the scores can be computed for all timesteps with a single forward solve (and backprop) of the ODE. This is not possible for DPS, since each conditional score must be computed sequentially, AFTER the previous score has already been applied to the trajectory. If DPS were to use the reviewer’s suggested strategy, it would require on average $n / 2$ extra NFEs per update, resulting in a sum total of 500K NFEs under the current parameterization. By comparison, our method achieves superior performance to DPS using as little as 1K NFEs.

Results are only provided for MNIST and FFHQ. We additionally provide experiments on ImageNet-1K, and demonstrate that our performance holds, with similar trends to Table 1.

Please discuss alternative approaches to posterior sampling with diffusion models, e.g., variational approaches (https://arxiv.org/abs/2305.04391) or resampling strategies (https://arxiv.org/abs/2307.08123). We thank the authors for bringing up these concurrent works. We have added them to the manuscript. We note that ReSample (Song et. al, 2024) suffers from the same approximation error as DPS when estimating $x_0$ via Tweedie’s. Conversely, RED-Diff (Mardani et al., 2024) indeed does not require Tweedie’s, but requires approximations to their variational framework (e.g., the stop-gradient) when computing their proposed variational loss, thus also yielding an approximate solution.

Hyperparameter choices for the baselines. For baselines from competing works in Table 1, we directly replicate the reported hyperparameters from the respective papers. For our baselines, please refer to Table 2 for the hyperparameter values. For the T = 20 result, we let T = 20 and let the number of iterations be 50. For insight into how we selected these values, please see Section D.3 and Tables 4, 5, and 6.

Assumptions for the theorems. The assumptions required for Theorem 4.1 are summarized in Equations 27 and 28. Similarly, Theorem 4.2 requires Equations 30 and 31. We have additionally clarified that we require $\alpha > 0$, and $\log p(y | x)$ to be twice-differentiable (which holds when $p(y | x)$ is Gaussian, as is the case in our settings). We take our theoretical results very seriously and would be happy to clarify any other ambiguities in the theorems.