Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements

1. The contribution seems incremental.
2. Intuition for designing the crafted measurements should be provided. A visualization of the measurements during the sample trajectory is also helpful for the readers to follow.
3. The method relies heavily on the crafted measurement. The result may be unstable due to different crafted measurements sampled from the reverse DDPM process.

In my opinion this paper is technically flawed, incoherent in some places and contains no review of the relevant and close literature.

• First, the presentation of the methodology starts with "We propose DPS-CM, which perform diffusion posterior sampling from $p(x_t | y_t)$ with Crafted Measurement $y_t$ belonging to another diffusion reverse trajectory $( y_t )^T _{t=0}$ instead of the vanilla input $y$". In this statement it is not clear at all what "posterior sampling" from $p(x_t | y_t )$ means. If it means running the SDE (5) with $\nabla \log p(x_t | y_t)$ then it is not justified anywhere why this would yield a valid approach for sampling from the posterior distribution $p(x|y)$ and actually, this is not the case. The explanation for the statement above is that the authors are aiming to sample from the joint distribution $p(x, y)$ and they consider the reverse diffusion that targets the same. The $y$ component of the reverse diffusion involves the score $\nabla \log p^{Y} _{t}(y_t)$ (I use the superscript $Y$ to insist that this score is very different from the learned, marginal, score. And the $x$ component of the diffusion uses the conditional score $\nabla \log p(x_t | y_t)$. This procedure yields a valid sample from the joint distribution $p(x,y)$ but not the posterior distribution $p(x|y)$ where the observation $y$ is fixed. I am thus failing to see how considering the score $\nabla \log p(x_t | y_t )$ yields a valid procedure for sampling from $p(x|y)$. As far as I am concerned this is a critical aspect that is not discussed nor explained.

• Now I turn to the score $\nabla \log p^{Y} _{t}(y_t)$. This score is very different from the pre-trained one yet the authors claim in the paper that "Besides, $(y_t ) ^{0} _{t = T}$ shares the same diffusion model $s _{\theta}(\cdot, t)$ with restoration trajectory $(x_t)^0 _{t=T}$ because $x$ and $y$ lie on close manifolds in most typical inverse problems." In my opinion this claim deserves a lengthy discussion since it is far from obvious that this is true. For example, when the observation is an image with missing half, we can safely assume that the underlying distribution of the observation is that of the images from the dataset with missing half. Thus, running the diffusion process with the true score $\nabla \log p^{Y} _{t}(y_t)$ will return images with a missing half, whereas using the pre-trained score returns complete images. The same holds for many common inverse problems.

• The main motivation of the paper is that the injection of high frequency information early on during the diffusion process can be harmful to the reconstructions. The claim is that this can be avoided by using instead $\nabla \log p(y_t | x_t)$. Now assuming that this is true, from the derivations in the paper it is shown that one, in this case, has to estimate $\nabla \log \mathbb{E}_{y_0 \sim p(y_0 | y_t)} [p(y_0 | \hat{x}_0) ]$. In the paper this is estimated using a DPS approximation but if we assume that we can actually sample exactly from the distribution $p(y_0 | y_t)$, we would actually (and ideally) use a Monte Carlo estimate with samples $y_0 ^1, \dotsc, y_0 ^N$ from $p(y_0 | y_t)$:

which is an average over "high frequency information" and is at odds with the claims of the paper. Using instead the DPS approximation does indeed involve low frequency information but this doesn't align with the theoretical justification of the paper.

• As mentioned in the paper, the "crafted measurements" are sampled from the forward process $p(y_{1:T} | y_0)$. As explained above, the way this is done in practice is flawed. I am failing to understand why the authors do not simply sample the observations from the forward process of the Diffusion? This also gives exact samples. I understand that sampling forward means that all the samples need to be stored in memory, but one can also sampled from this distribution backwards (and exactly) by noting that $p(y_{1:T} | y_0) = p(y_T |y) \prod_{t = 1} ^{t-1} p(y_t | y_0, y_{t+1})$. Also the step 10 still involves the DPS approximation. This is again at odds with the justifications provided in the paper.

• Finally, the literature review in this paper is quite poor in my opinion. There are many papers in the literature that use the noised observations $y_{1:T}$ within the backward process. A few examples are listed below. The authors compare to [3] but the similarities are never discussed.

[1] Lugmayr, Andreas, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. "Repaint: Inpainting using denoising diffusion probabilistic models." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11461-11471. 2022.

[2] Trippe, Brian L., Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. "Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem." arXiv preprint arXiv:2206.04119 (2022).

[3] Dou, Zehao, and Yang Song. "Diffusion posterior sampling for linear inverse problem solving: A filtering perspective." In The Twelfth International Conference on Learning Representations. 2024.

1, The proposed crafted measurement $y_t$ in lines 3–6 of Alg1. is based on an underlying assumption that the measurement sample lies within the domain of the deep score network $s_\theta$. However, this assumption may not hold for various inverse problems, such as CT/MR imaging and phase retrieval, where the measurement distribution often differs from that of the training images $x_0$. This discrepancy may fundamentally limit the practical contribution of the DPS-CM method, as it is unclear how effective the approach would be if the measurements differ significantly from the training data distribution.

2, The numerical comparisons provided may not accurately or fairly represent the performance of the baseline methods. For example, in Table 5 in Appendix A, hyperparameters for DPS-CM are carefully fine-tuned, whereas the baselines use their original settings. Given that the experimental setup does not strictly follow the configurations of each baseline, the reported performance gains for DPS-CM may appear more significant than they truly are.

3, The connection between lines 3-6 in Algorithm 1 and the estimation of the posterior mean $\mathbf{E}[y_0 | y_t]$ is not clearly explained. It remains unclear how the update for $y_{t-1}$ is theoretically derived. Providing additional clarification would enhance understanding of the theoretical basis for this update.

4, Although the idea of moving from “low-to-high” frequency for improved sampling is interesting, the proposed approach ultimately employs a relatively simple convex combination of the likelihood $p(y|x_t)$ with a fixed measurement $y$. This limits the contribution to a primarily engineering-oriented approach, lacking deeper theoretical connections as the paper intially suggested. Additionally, the extra hyperparameters introduced make the algorithm more challenging to fine-tune.

Section 3 raises some questions that, I hope, the authors can help clarify.

First, could the authors clarify the difference between $y_t$ in line 291 (noisy measurements) and $\text{y}_t$ as defined in line 295 (crafted measurements)? As I understand it, noisy measurements are sampled from the forward trajectory of the $\\{y_t\\}$ diffusion process, while the crafted measurements are generated during the reverse-time trajectory. Does the direction of the diffusion impact the definition of these $y_t$ values?

Additionally, could the authors please clarify why $E[y_0 | y_t] = y$ in line 285, but $E[\text{y}_0 | \text{y}_t] = \hat{\text{y}}_0$ in lines 303-304?

I also have a concern regarding the equation in lines 306-307. Consider the scenario where, in the equation in line 126, the operator $A(\cdot)$ is identity, and noise $\eta$ is constant zero. In other words, $y = x_0$. Also, assume that $p(x_0 = 0) = 1$, so $x_0$ is a constant, specifically $x_0 = 0$. Under these conditions $y = x_0 = \hat{y}_0 = \hat{x}_0 = 0$, and consequently the right-hand side of the equation in lines 306-307 is constantly 0.

Now let us demonstrate that the left-hand side of the equation can be arbitrary far from zero. Assume that $x_t = x_0 + n_x = n_x\ ,$ and $y_t = y_0 + n_y = n_y\ .$ Additionally, suppose that $n_x$ and $n_y$ are Gaussian-distributed with $n_x \sim N(0, 1)$ and $n_y|n_x \sim N(n_x, 1)\ .$ Then we have $p(y_t | x_t) = (2\pi)^{-0.5}\exp\\{-0.5(y_t - x_t)^2\\}$, which yields $\nabla_{x_t}\log p(y_t | x_t) = y_t - x_t$. Since both $y_t$ and $x_t$ are unbounded, the left-hand side of the equation in lines 306-307 is also unbounded, and can be arbitrary far from zero.