Restorer Guided Diffusion Models for Variational Inverse Problems

There are several claims in the paper that are misleading, wrong, or not useful.

1-1. Figure 1 does not make a lot of sense to me. The inverse problem is defined to have a measurement likelihood $p(x|y)$, and there cannot be two more classes to these problems. For instance, [1] belongs to the family of blind inverse problems, and you cannot call them generative, just because the parameters of the forward model is estimated from a generative model. That has nothing to do with the definition of an inverse problem. It is how you solve it.

1-2. (pg 2, paragraph 1) "However, these methods remain in the paradigm of the measurement-based likelihood": why do you separate (a) and (b) in Fig. 1 then?

1-3. restoration is related to $p(x|y)$, while the measurement process is $p(y|x)$. You cannot define the measurement process through a restoration model.

2. (pg 2, paragraph 2) "beyond the restriction of deterministic deterioration process without any extra training...": You do need extra training, as you need a restorer for guidance. This is not any different from learning different bridges for specific degradation models such as IR-SDE.

3. Eq (7) is not useful. The essence of posterior sampling is to estimate $p(x_0|y)$, which is a component of the integrand. If you are modeling $p(x_0|y)$ with a Gaussian distribution with mean centered on $\mathcal{R}(y)$, then it means that your posterior samples will be the predicted estimates from your restorer plus some AWGN.

4. The method is un-related to classifier-free guidance. In order for CFG to correspond to sharpening the posterior distribution with the parameter $\omega$, the multiplicative constant should remain $1 - \omega$ and $\omega$. Introducing new parameters with $\eta, \rho$ can simply be considered a single parameter by defining $\eta/\rho$. This is no different from how the gradient of the log likelihood is weighted in DPS.

5. Because Eq (7) is not useful, and because of the point made in 3, the method presented in Eq (11) simply boils down to reverse diffusion constrained to be close to $\mathcal{R}(y)$. I can understand that this might boost the performance of the restorer a little bit by leveraging the pre-trained generative prior of the score function, but with extensively more computational burden.

6. Overall, the method is not grounded either theoretically or intuitively.

7. The only really useful comparison is against NAFNet and IR-SDE. All other comparisons are not useful. The method should compete against state-of-the-art supervised methods, as it leverages the pretrained restoration model itself.

References

[1] Chung, Hyungjin, et al. "Parallel diffusion models of operator and image for blind inverse problems." CVPR 2023.

While the paper proposes to go beyond analytical and deterministic forward operators, it needs pretrained task-specific restoration networks. On the other hand, despite using state of the art supervised task-specific restoration networks, the proposed approach provides only a marginal improvement over this end-to-end trained network. Even the supposed benefits in terms of out of distribution robustness do not seem to be significant when a strong restoration network such as MPRNet baseline is used.

The intuition behind using \mathcal{R}(xˆ0|t) in step 6. of algorithm 1 and 2 is not clearly described. It is more intuitive for restoration network to take measurement or an estimate of it as input. Section 3.4 Restorer traveling states this step as optional.

It is not clear why a least squares similarity between a noisy iterate and the degraded measurement (3rd term in equation 15) referred to as measurement boosting should be included. This implies the measurement is highly similar to the solution as t->1, which is not necessarily true. The ablation study in section 4.2 shows that including restorer traveling and measurement boosting improves performance. Yet, why these should work is not explained.

Furthermore, there are tasks where x and y have different dimensions, for example super-resolution, compressed sensing where such terms cannot be used.

---

Issues with related work (section 5): Ulyanov et al.(2018) use untrained neural networks as priors, this approach is not same as deep generative priors.

A recent work also uses a pre-trained restoration network in the context of diffusion-based restoration. The authors could discuss this work: PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance https://arxiv.org/abs/2309.10810

---

Terminology and language used in the paper is strange and not consistent with what is used in the literature. The language makes it a hard read, requiring several attempts to understand what is meant. Writing needs a lot of improvement.

The sense in which the term "variational" is used in this paper, is very different from its use in literature. The paper uses this to mean unknown and random measurement process.

"manual destruction" instead of real world blur (last line in page 1) "realistic" instead of realism (multiple instances throughout the paper)

"which release the deficiency of the deterministic deterioration process with bestowed variational capability" paragraph3 in introduction. It is not clear what this means.

Step 2: Restorer traveling in page 5, "to release the great potential" --> to realise the great potential

It is not clearly described what is a "cluster process" or "augmented cluster process, such terms are used repeatedly in the paper.

Sample typos/mistakes : "which similar to the classifier guidance" "and impotent to variational unpredictable" ", which prone to lead the " "which prone to lead the suboptimal prototype-biased solving result" "where we remain the weighting parameter"

• Despite overall clarity, the paper contains important shortcuts and tends to strongly dismiss the relevance of concurrent works. These include traditional methods (that are weakly reviewed, e.g. [1,2,3]) or do not mention important recent works as [4, 5].
• Similarly, the paper does not seem to perform fair baselines comparisons.
• While the paper is aimed at general inverse problems, experiments are only performed on very specific inverse problems that are of limited relevance in practice.
• Similarly, the proposed method does only seem to apply when measurements and target images live in the same domain, which is a very limiting assumption in inverse problems and prevents from using this method in most real imaging setups. There is thus a mismatch between the claimed generality of the proposed method and its real applicability setup.

References

[1] Bredies, Kristian, Karl Kunisch, and Thomas Pock. "Total generalized variation." SIAM Journal on Imaging Sciences 3.3 (2010): 492-526.

[2] Mallat, Stéphane. A wavelet tour of signal processing. Elsevier, 1999.

[3] Chambolle, Antonin, and Thomas Pock. "A first-order primal-dual algorithm for convex problems with applications to imaging." Journal of mathematical imaging and vision 40 (2011): 120-145.

[4] Romano, Yaniv, Michael Elad, and Peyman Milanfar. "The little engine that could: Regularization by denoising (RED)." SIAM Journal on Imaging Sciences 10.4 (2017): 1804-1844.

[5] Zhu, Yuanzhi, et al. "Denoising Diffusion Models for Plug-and-Play Image Restoration." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

1. There is a disconnect between the paper's stated aim and the approach it employs. While the paper claims to address problems involving varying degradation processes, the methodology it presents essentially relies on invoking a pre-trained model designed for general image recovery to handle varying degradation. Consequently, the paper lacks a specialized design tailored to address the unique challenges posed by unknown and varying degradation process. In essence, the same protocol working on existing image restoration diffusion models by simply running a diffusion-based denoising network to post-process the output from an image restoration method. This calls into question the novelty of the proposed approach.

2. The paper replies on a pre-trained model for general image recovery, which is already equipped to handle varying degradation. This approach essentially sidesteps the central challenge of providing a conditioned posterior distribution based on measurements, a crucial aspect when extending the problem from image denoising to more complex image recovery tasks that involve a non-trivial null space. Consequently, the paper does not appear to make a substantial contribution to advancing the state of the art in this specific problem domain.

3. The mathematical derivations put forth in the paper primarily address linear inverse problems with known measurement matrices. This focus is incongruent with the complexities of image recovery tasks that involve unknown degradation processes, many of which cannot be formulated as linear inverse problems. Given that these derivations are already well-documented in existing literature, the paper does not make novel theoretical contributions.

4. The experiments have many limitations.

4.a. The paper's performance is highly contingent upon the pre-trained restoration model it employs, referred to as the "restorer." The authors utilize pre-trained models with published demonstration on other tasks such as denoising and super-resolution, it should be noted that while these models claim to be suitable for general image restoration, their performance varies across different tasks. The experiments should be conducted on either the same image tasks showed in the paper they cited. Or adopting some more universally effective model, such as "Restormer,"

4.b. The paper lacks crucial details, including the datasets used for training both the diffusion and restoration models. The characteristics of these datasets are essential for evaluating the generalization performance of the proposed method, yet this information is absent.

4.c. This paper is more a applied paper. From this viewpoint, the range of models used for comparison is overly narrow. The experimental setup primarily resembles ablation studies rather than comprehensive evaluations against state-of-the-art methods for each image task. Consequently, the experiments fall short of demonstrating the method's effectiveness in comparison to leading approaches in the field.