Solving Linear-Gaussian Bayesian Inverse Problems with Decoupled Diffusion Sequential Monte Carlo

Thank you for the comments, which have been of great value to improve the paper.

Design choices in SMC make big difference in practice, motivating us to construct a new and better algorithm

The reviewer correctly points out that TDS and MCGDiff already enjoy asymptotic exactness. However, the choice in intermediate targets and in proposal can make a big difference in practice with finite number of particles. Motivated by previous works on SMC for diffusion priors, we set out to design a new and improved algorithm. The different SMC algorithms indeed show very different performance in the experiments, with DDSMC outperforming both MCGDiff and TDS, providing strong evidence that the design choices made in DDSMC give better practical efficiency compared to the other SMC methods. As a reply to "By adjusting annealing parameter $\eta$, this new prior can generalize previous SMC based methods" we would like to emphasize that DDSMC is a novel SMC method for the problem under study regardless of the annealing parameter $\eta$. In fact, our main contribution is the development of this novel SMC method, and we view the generalization of the DAPS prior (i.e., introducing $\eta$) as a secondary contribution. See also the reply to psuz. See response to tiGx and XuMA regarding asymptotic exactness

We will rephrase the part about PF-ODE for sampling from $q(x_0|x_{t+1})$

In the background section (line 104 col 1) we wrote that it is possible to use PF-ODE to sample from $q(x_0|x_{t+1})$, in order to generate a sample trajectory from the prior. We thank the reviewer for pointing out this error, and we agree that this is incorrect

Note that this paragraph was only included as an "intuitive explanation" of how the proposed method works, and the validity of the method does not in any way rely on the PF-ODE sampling from $q(x_0|x_{t+1})$. What we actually meant to say with this section was that, conceptually, a (convoluted) way to simulate from the prior backward process would be: Initialize $x_T\sim q(x_T)$. For $t=T-1,\dots,0$,

1. Solve $\hat x_{0,t+1}=\text{PF-ODE}(x_{t+1})$ from time $t+1$ to 0
2. Sample $x_{t}\sim q(x_{t}|x_0=\hat x_{0,t+1})$

This would result in samples such that, marginally for any $t$, $x_t\sim q(x_t)$, but we do not get samples from the joint $q(x_{0:T})$, nor from the conditionals $q(x_0|x_{t+1})$ as the reviewer correctly points out. This sampling process motivates the DAPS prior that we use (and generalize), which is why we mentioned it in the background, but we will of course make sure to update the text so that it is mathematically correct when revising the manuscript.

We have evaluated DDSMC on protein structure completion

The reviewer writes "The evaluation of this paper is a kind of limited." In the response to XuMA we have added results for another experiment concerning protein structure completion, showing that DDSMC can outperform the tailored APD-3D method out-of-the-box in (realistic) high-noise setting.

We have now evaluated using 1k images

We reran the experiments on 1k images, with essentially identical results (see response to psuz). We also computed PSNR, where now DDRM is the overall strongest model. However, just as for LPIPS, the standard deviation is rather large. Given that our method aims to recover posterior distributions, PSNR as a per-pixel metric (even stricter than per-sample metric like LPIPS) does not represent an ideal metric here, and for generative models there is often a trade-off between perceptual (e.g., LPIPS) and distortion (e.g., PSNR) metrics [1]. We will supply the PSNR table in the appendix with a comment in the main paper. We don't compute FID as we are focused on sampling from 1k different conditional distributions and not 1 unconditional distribution.

[1] Blau and Michaeli, The Perception-Distortion Tradeoff, CVPR 2018

Clarification regarding complexity

We agree that a discussion is missing and will add this in the paper. In summary: DDSMC-Tweedie requires N times more (N=number of particles) NFEs per diffusion step compared with DDRM, and DDSMC-ODE has N times the NFE of DAPS. DDSMC-Tweedie has the same complexity as MCGDiff and requires slightly fewer NFEs than TDS (which requires differentiating through the score-function). See response to XuMA for an additional study using fewer particles to obtain the same number of NFEs for DDSMC-ODE as MCGDiff in the GMM case. For images, we are already using fewer NFEs as we are using fewer particles.

The additional NFEs required for SMC should be viewed as way of trading off improved sample quality with compute. As seen in the additional GMM experiments with fewer particles, this aspect holds empirically as the performance improves when using more particles (especially over using a single particle). For methods like DDRM or DCPS, we have attempted to use as much computation as reasonably possible and they still fail while DDSMC effectively enjoys the compute-quality trade-off.

Thank you for taking the time to read and comment on our paper, which certainly has been useful to make the paper better. We answer concerns and questions below.

Assumptions in proposition A.1 concerns DAPS vs standard prior, not asymptotical exactness of DDSMC

Thanks for pointing out the unclarity regarding the assumptions in Proposition A.1., and how they affect the consistency of DDSMC. We emphasize that Proposition A.1 only concerns properties of the DAPS prior, not the DDSMC algorithm per se. Specifically, it refers to whether the DAPS prior ($\eta =0$) and the standard diffusion prior ($\eta=1$) result in the same (marginal) prior $p_\theta(x_0)$. If the assumptions do not hold (as they will not in practice), these priors are different. However, regardless of the choice of prior (we view this as a design choice, more about this below), DDSMC will have asymptotic exactness guarantees, i.e., the empirical approximation will converge to the corresponding posterior induced by the chosen prior. It is in this last point where SMCDiff and FPS are different: they also start from a diffusion-model prior which (combined with the given likelihood) induce a posterior. However, these algorithms then target an approximation of the induced posterior. This approximation, which is the target of their respective SMC samplers, will correspond to the actual induced posterior only when the forward and backward kernels match. This is what we mean by "SMCDiff and FPS [...] are therefore not consistent in general".

As mentioned above, it's important to note that the prior in DDSMC is a design choice. We have developed the method based on a generalization of the DAPS prior, because the decoupling offered by this prior has proven to be useful in prior work. However, the DDSMC method is equally applicable to the "standard diffusion prior" (simply set $\eta=1$) and the algorithm will then provide consistent approximations of the posterior induced by this prior. In this case we do not rely on Proposition A.1 at all.

We thus believe that the assumptions in Proposition A.1 are used in a fundamentally different way in DDSMC than in SMCDiff and FPS. We thank the reviewer for highlighting this important detail, and we will make a clarification around line 144 col 1 in a revised version of the paper. See also response to reviewer tiGx about asymptotical exactness. If there are any more questions or concerns about any of this, we are happy to answer in a follow-up comment.

We have tried GMM experiments with same compute budget as for MCGDiff

Using Tweedie's formula as the reconstruction requires just a single evaluation of the score function, meaning in this case we are already using the same compute budget as MCGDiff in the GMM experiments. We will clarify that this is what we mean in line 358 col 1 when saying that "DDSMC outperforms all other methods, even using Tweedie's reconstruction".

We do agree, though, that using DDSMC-ODE under same compute budget is also interesting, and have therefore performed additional experiments using DDSMC-ODE with 12 particles (20x less than the other methods) and 25 particles (10x less, as the number of steps in the ODE decreases from 20 to 1, i.e. $\sim 10$x more score evaluations/particle on average), and the results show that in low dimensions, 12 or 25 particles is still better than the Tweedie (and hence, MCGDiff/TDS), but this changes for higher dimensions, where using Tweedie with more particles seems to be the better choice. This shows how the SMC aspect (multiple particles and resampling) indeed is an important aspect in the performance. We will add these results in the appendix, along with a similar experiment where we change the number of ODE-steps instead, and a comment in the main paper.

Additional empirical results

We agree with the reviewer that "an addition of a different source of real data would greatly enhance the evaluation", and therefore looked at the protein structure completion in ADP-3D [1]. Their method is built for that type of task, and our method performs well out of the box, outperforming ADP-3D on higher, but realistic, noise levels (where we had to tweak their learning rate to give reasonable results).

RMSD on 7qum protein. Columns indicate that every $n$ residues are observed.

[1] Levy et al. Solving Inverse Problems in Protein Space Using Diffusion-Based Priors, arXiv, 2024

We thank the reviewer for taking the time to comment on our paper. We have tried to address all your comments below, but there were a few points that we did not quite understand, so we kindly ask you to clarify if we in fact misunderstood some of your points.

Merging SMC with decoupled diffusion is our core contribution

As the reviewer comments "algorithms that merge SMC with decoupled diffusion in an innovative manned could have strengthened the paper", we want to highlight that this is exactly the core contribution that we make in our paper! As far as we are aware, this is the first time that SMC is merged with decoupled diffusion, and we are therefore not sure what this comment refers to. We are happy to answer any follow up clarifications on this.

The reviewer comments that "The proposed DDSMC method reduces to existing methods in extreme cases (e.g., inverse temperature eta = 0 and using PF-ODE for reconstruction)." We do not agree with this claim since it misses the point that our core contribution is an SMC algorithm building on the (generalized) DAPS prior. Hence, this claim would only be correct if we also restrict DDSMC to using a single particle, but the "parallel particles" are a key ingredient in SMC. Running an SMC algorithm with a single particle corresponds to sampling from the proposal, which if using $\eta=0.0$ and PF-ODE would be more or less equivalent to DAPS (if instead using $\lambda_t^2=\sigma_t^2$ in eq (16)). To verify that, we ran DDSMC-ODE on the GMM case with a single particle, and the numbers obtained with $\eta=0.0$ are essentially identical to DAPS (differing by at most 0.03 from each other). Hence, we can conclude that the introduction of the SMC framework is the key ingredient that leads to the improved performance over DAPS. We will incorporate these results in the appendix, and make a comment in the experiment section in the main paper. Next, we do agree that a further analysis of the number of particles would be beneficial, and as part of the response to XuMA, we performed additional experiments with fewer particles using the DDSMC-ODE.

We target posterior sampling, which is verified in the GMM task

We agree that our experiments largely depend on synthetic experiments. However, we believe that this is a necessary sanity check for a method such as DDSMC which is designed to target the correct posterior distribution, intended to empirically prove that our proposed technique samples from the true posterior. In order to achieve this, having an exact score/ground truth posterior is necessary to control errors.

The purpose of the image experiments is to first show qualitatively that our method works also in high-dimensional, real-world problems, and we included the LPIPS metric to also show this quantitatively: we are in line with SOTA methods, and outperform MCGDiff, which is the closest comparable method to DDSMC.

We emphasize that our focus is on recovering the posterior distribution, and we do not aim to improve the per-sample quality like DDRM, but instead, the population quality. However, a problem is that most image-related tasks lack the ground truth posterior, and even a good approximation thereof. To our knowledge, there are no principled metrics to gauge such performance. As such, the synthetic experiment is the only setting we can rely on to compare the methods' posterior sampling ability in a principled way. We revise our Experiment section to reflect on this.

We have evaluated DDSMC on protein structure determination

The reviewer comments that "The main limitation lies in the experiments." In the response to Reviewer XuMA we have added results for another experiment concerning protein structure determination. We show that DDSMC can outperform the tailored APD-3D method out-of-the-box in (realistic) high-noise scenarios. Please see response to XuMA for further details.

We can see clear effects of changing the inverse temperature $\eta$

As mentioned above, we view the DDSMC method itself as our primary contribution, and the generalization of the DAPS prior (such that we can interpolate between DAPS and the standard diffusion prior using the inverse temperature $\eta$), as a secondary contribution. The reviewer comments that we should demonstrate "how varying $\eta$ affects the result and that an intermediate value between 0 and 1 offers better performance". We agree with the observation that $\eta=0.0$ is the best in high dimensions for DDSMC-ODE, but we emphasize that $\eta=0.5$ is better for DDSMC-Tweedie in this case, and is also the best choice in lower dimensions for DDSMC-ODE, while $\eta=1.0$ seems to be better for DDSMC-Tweedie in lower dimensions. It hence certainly seems to be an interplay between the dimension of the data, the choice of reconstruction function, and the choice of $\eta$. We already discuss this in line 304 col 2 and onward, but will extend the discussion in a revised version of the paper.

We thank the reviewer for their comments, and we address their concerns below (see answer to zGWq regarding complexity and NFEs).

We will clarify the asymptotical exactness guarantees

SMC provides consistent approximations of its sequence of targets $\lbrace \pi_t(x_{t:T})\rbrace_{t=0}^T$ under weak conditions. However, since we only care about the final target, it is enough that $\pi_t(x_{0:T})$ admits the true posterior $p_\theta(x_0|y)$ as a marginal for the method to be consistent. This is the case for DDSMC by construction; see Eq (7) with $t=0$. The intermediate targets $(\pi_t(x_{t:T}), t>0)$ as well as the proposals can be seen as design choices that affect the efficiency of the algorithm but not its consistency. The particular ways in which we design these quantities (while ensuring correctness of the final target) constitute the DDSMC framework. We mention this general fact in line 92 column 2 regarding the intermediate targets but will add a comment regarding the proposal. We note that the specific questions that you ask regarding approximations in Eq (13), (16), (68) only affect the intermediate targets and/or proposal and thus not the consistency of the final target approximation.

We realize that this should be further clarified, and we will hence add a "theorem-like" paragraph, making it clear how our algorithm will be asymptomatically exact. See also response to XuMA.

We view reconstruction function $f_\theta$ as a design choice

In our experiments, we tried two types of reconstruction functions: either using Tweedie's formula (see line 87 col 1) or the PF-ODE. As we view this as a design choice, we have not explicitly written how $f_\theta$ is computed in Algo 1. It is true, however, that if using the PF-ODE, this requires an inner loop. In eq (66) there is a typo, where $t$ should be replaced by $t'$ (the inner loop time variable), and in line 323 we mean that we start at $x_{t'} = x_t$, then use Eq 66 for $t'=t-1, t-2, \dots, 0$. I.e., we start from a sample at the diffusion time (outer loop index) $t$, then convert that into a sample $x_0$ using as many steps in the inner loop as there are "left" in the outer loop. We will fix these typos and clarify line 323.

This flexibility also affects the computational cost/NFE, where the Tweedie reconstruction avoids the outer loop and is therefore more efficient. Compared to MCGDiff and TDS, DDSMC-Tweedie has the same number of NFE (but avoids the expensive differentiation of the reconstruction network in TDS), but can still provide better result. The DDSMC framework is general and choice between Tweedie and PF-ODE (or some other reconstruction method) becomes a computational trade-off for the user.

While falling under the same framework as TDS and MCGDiff, we provide better empirical performance

Both TDS and MCGDiff are SMC methods, and we differ in how we design both the proposal and targets, which empirically shows improved performance. Additionally, we make a secondary contribution in the choice of prior (using $\eta\neq 1$), which can further improve performance.

We evaluate DDRM with fewer steps, gives worse performance

We used 300 steps when running DDRM to match that of DCPS, but on the reviewers advice we also tried 20, which shows worse result. We will add that to the appendix. For the GMM case we used 1k steps as more steps would mean less discretization error, and instead using 20 shows worse performance for 8 and 80 dimensions, while being similar at 800 dimensions.

We have now evaluated on 1k images, computed standard deviations

We took the reviewer's advice and evaluated on a 1k image validation set. The numbers, however, differ only by a maximum 0.01. The standard deviations of the 1k values ranges between 0.01 up to 0.075.

We are not aware of linear inverse problems without implementable SVDs

We make it very clear that this method tackles linear inverse problems (it is in the title) which is still an open question (e.g., MCGDiff and DCPS tackle exactly this). However, we are not aware of linear inverse problems which do not have implementable SVDs and would gladly appreciate pointers to examples of this if implementable SVDs is a large concern.

Other points

• We will add a paragraph in Related work to cite the suggested papers and discuss related MCMC methods.
• We agree that the expression for $p_\theta(x_t|x_{t+1})$ is a definition in the current problem formulation (although based on approximating $q(x_t|x_{t+1})$ when training the generative model) and will clarify
• In the resampling step, a set of "ancestor indices" $`{`a_t^i`}`_{i=1}^N$ are sampled from the Multinomial distribution, and each particle $x_t^i$ is then replaced by $x_t^{a_t^i}$. The circular dependency comes from overloading the notation. We will clarify to avoid confusion
• The notation in F.2.1. came from following the model formulation used by DAPS, which gave rise to a notational inconsistency. We will make sure to clarify