Stochastic Deep Restoration Priors for Imaging Inverse Problems

We thank the Reviewer for their time and feedback. Please see below for our point-by-point responses to your comments.

Comparison to Diffusion-Based Methods: The paper lacks a comprehensive comparison to diffusion-based methods, such as DiffIR [1] and DDRM [2]. Including these comparisons would strengthen the results and provide clarity on how ShaRP performs relative to other leading methods in the field.

Thank you for bringing these two methods to our attention. We have cited corresponding papers and included comparisons with these two methods in Section D.3 of the supplement.

Supervised vs. Self-Supervised ShaRP: In line 154, the authors mention a key contribution: "We implement ShaRP with both supervised and self-supervised restoration models as priors and test it on two inverse problems: CS-MRI and SISR." However, the experimental section does not provide a direct comparison between the self-supervised and supervised versions of ShaRP for the same task. This comparison is necessary to assess the benefits of the self-supervised approach.

Prompted by your comment, we revised the sentence to eliminate any confusion and added a table to clearly illustrate the performance difference between both versions of ShaRP. It is important to emphasize that the key advantage of the self-supervised approach is not superior performance, but the ability to learn without requiring fully-sampled MRI data—a scenario where training a Gaussian-denoising network is not feasible.

Self-Supervised Nature: For a restoration network to be trained on a set of tasks, such as a set of blur kernels (H_i), access to ground truth data is still required. This approach, which involves sampling multiple times from the ground truth, raises questions about whether the method can truly be considered self-supervised. Clarification is needed regarding the self-supervised claim.

Prompted by your comment, we have clarified the term “self-supervised”. Our approach trains an MMSE restoration network using only undersampled measurements, without any access to the clean images. Specifically, for 8x undersampled MRI restoration, we use two 8x undersampled measurements, one as the training target and another as the input to the restoration network. Note that the combination of these two measurements remains undersampled, ensuring no reliance on fully-sampled data. This terminology is well-established in the CS-MRI literature, as can be seen in references [1–3] below.

[1] Millard, Charles, and Mark Chiew. "A theoretical framework for self-supervised MR image reconstruction using sub-sampling via variable density Noisier2Noise." IEEE transactions on computational imaging (2023).

[2] Gan, Weijie, et al. "Self-supervised deep equilibrium models with theoretical guarantees and applications to MRI reconstruction." IEEE Transactions on Computational Imaging (2023).

[3] Akçakaya, Mehmet, et al. "Unsupervised deep learning methods for biological image reconstruction and enhancement: An overview from a signal processing perspective." IEEE Signal Processing Magazine 39.2 (2022): 28-44.

Use of Multiple Degradation Operators: The rationale for using a set of degradation operators (H_1, H_2, \ldots, H_k) in cases where the target problem involves only a single fixed operator (e.g., (H_1)) is unclear. It would be helpful if the authors could explain why introducing multiple degradation operators is necessary or beneficial when solving a fixed-task problem.

1. The strength of our framework lies in its versatility, enabling seamless integration of a wide-range of restoration models within a unified formulation in eq. (6). Note how our approach achieves performance improvements over popular methods using Gaussian-denoising priors.
2. It is worth highlighting that without our framework, the direct application of a restoration model trained for $8\times$ MRI reconstruction to the $4\times$ and $6\times$ scenarios results in a significant performance degradation. This is shown in Section C.2 of the supplementary material. Our framework enables a principled integration of the 8x model as prior without re-training and without performance degradations.

Thank you for your detailed responses and for addressing the concerns raised. After reviewing the other reviewers' comments and your rebuttals, I have some additional thoughts.

Firstly, I concur with reviewer XTiX's observations regarding the argument about the inadequacy of Gaussian-denoiser networks for solving general inverse problems, at least from a theoretical standpoint. A Gaussian denoiser is sufficient for learning the image prior, with the data fidelity term addressing noise based on the corruption in measurements. Additionally, to ensure the learned prior is independent of the forward operator and generalizable, a Gaussian denoiser is preferable to a restoration network tied to specific degradation operators.

In this way, it appears that the key novelty in your work is the use of multiple degradation operators. While this approach adds value and has potential benefits, I am not entirely convinced by the arguments presented against the use of Gaussian denoisers in this context.

Thus, I prefer to maintain my original rating of 6.

We disagree with your assessment of our paper as a "trivial extension" of Hu et al. (2024c) and with the argument that we “directly copied” a figure from Hu et al. (2024c).

None of the figures in our submission are copies from any other paper. All the figures represent different experiments, datasets, and analyses involving our method, and of course they are not included or related to Figure 1 in Hu et al. (2024c) or any other figure in previous works. We respectfully request the reviewer re-read both papers more carefully and withdraw the baseless accusation. We take unfounded accusations of plagiarism very seriously, and the reviewer should too.

ShaRP is a Non-Trivial Extension:

1. New regularizer: The regularizer in ShaRP is different from that in Hu et al. (2024c) because it promotes solutions whose multiple degraded observations resemble realistic degraded images. This leads to a richer image prior that integrates information from diverse degradation types, leading to more robust restoration capabilities.

2. New algorithm: (a) ShaRP is a stochastic algorithm, while DRP from Hu et al. (2024c) is deterministic. Saying that ShaRP is a trivial extension due to the use of restoration operators is analogous to saying that SGD is a trivial extension of the gradient descent due to the use of the gradient. (b) ShaRP eliminates the need for a scaled proximal operator used DRP from Hu et al. (2024c). Thus, unlike the algorithm DRP, ShaRP doesn’t need the assumption that $H^TH$ must be positive definite, making it applicable to many more degradation operators.

3. New theory: Our Theorem 1 is completely novel. It shows that our novel regularizer can be efficiently minimized using SGD. Hu et al. (2024c) doesn’t have any analogous result.

4. Training from undersampled measurements: ShaRP doesn’t require fully sampled MRI measurements for training, which is not possible using the DRP method in Hu et al. (2024c). This flexibility is an essential for MRI and is a significant breakthrough not achieved by any previous method.

5. Much better performance: ShaRP outperforms DRP from Hu et al. (2024c) across all experiments. ShaRP is on average 3.65 dB better than DRP on CS-MRI in Table 1 and 0.5 dB better than DRP on super-resolution in Table 3.

We hope that this clarification addresses your concerns.

Sincerely, The Authors

We thank the Reviewer for their time and feedback. Please see below for our point-by-point responses to your comments.

The presentation of the paper could be improved in some places. In particular, I am missing a more concise introduction to the motivation behind the actual concept of ShaRP in Section 4. The parameter $b$ representing the number of restoration problems is essential for the numerical performance. However, I am lacking a substantial discussion of the role of this parameter. In particular, no details about the role of $\alpha$ (see Section 5.1) and its impact on the results were provided.

We have edited the manuscript to improve the presentation. The essence of our framework is captured by the regularizer in eq. (6), which is maximized if degraded versions of $x$ are highly probable in the distribution $p(s|H)$, where $H$ is sampled from $p(H)$. Hence, mathematically our regularizer can incorporate infinitely many degradation operators. While adding the ability to restore from more degradations improves the performance, it increases the computational complexity of training the restoration network. The revised manuscript includes a new discussion and an ablation study in the supplementary material showing the impact of $b$.

Can you please provide more details about the impact of the choice of $\alpha$ on the numerical results? An ablation study might help here. Likewise, you could numerically evaluate the impact of $b$.

The revised manuscript includes the requested ablation studies in Section E.2 (see also below).

What happens if you modify the number of restoration priors in B.1.1?

We added an ablation study to Section E.2 on the performance under different values of b.

Please rewrite the motivation in Section 4. In particular, I am missing a good motivation and reasoning for the actual definition in equation (6), which might help the reader to better understand ShaRP. In addition, the object $G_\sigma(s - Hx)$ could be better motivated from a mathematical point of view.

We have edited the corresponding section of the paper. Our regularizer is minimized if degraded versions of $x$ are highly probable in the distribution $p(s|H)$, where H is sampled from $p(H)$. In other words, a solution x is considered good if, for all considered $H$, its degraded version $Hx$ matches the degraded versions $Hx^*$ of clean images $x^* \sim p(x)$.

The results in Table 4 do not show a clear tendency for DiffPIR, DRP, or ShaRP. Is there a reason for this available? Are there specific conditions under which each method performs best?

Prompted by your comment, we included a more detailed discussion for this table.

The table shows that ShaRP achieves the best performance in terms of distortion metrics such as PSNR and SSIM, while DiffPIR excels in the perceptual quality metric LPIPS. This trade-off arises because DiffPIR operates as a generative method that inherently favors perceptual quality. ShaRP consistently outperforms DRP in both distortion and perceptual metrics, showing the advantage of using a prior based on multiple degradation operators.

Thanks for the precise answers and the clarifications in the paper. I appreciate your feedback, but I am not enthusiastic about the manuscript, which is why I do not change my evaluation. One major reason for this judgement is the incremental novelty in this field (e.g., in comparison with Hu et al 2024c) as also outlined by two other reviewers.

We thank the Reviewer for their time and feedback. Please see below for our point-by-point responses to your comments.

I am not particularly convinced about the motivation for using a general restoration prior, that arises out of training a deep reconstruction operator for linear inverse problems (with several degradation operators), for solving another linear inverse problem. While learning Gaussian denoisers is a problem that is fundamentally easier than solving a general inverse problem (with an ill-posed operator), this is not the case here.

Prompted by the feedback, we have revised the paper to better motivate our approach. An important point often overlooked in the literature is that Gaussian-denoiser networks are suboptimal when used as priors to restore other, non-Gaussian artifacts. Our framework allows for tailoring the prior to the artifacts of a given inverse problem, providing an advantage over Gaussian-denoising networks. Our experiments show that restoration networks trained to remove MRI of artifacts at $8\times$ can serve as effective priors for MRI reconstruction at $4\times$ and $6\times$ accelerations. Additionally, our framework remains compatible with Gaussian-denoising networks, as they can always be incorporated into our prior using $H = I$.

The novelty in the convergence analysis (namely, the proof of Theorem 2) is unclear to me. To my understanding, the assumptions and the result are along the same lines as in the paper entitled “A Guide Through the Zoo of Biased SGD” by Demidovich et al. (which the authors have cited).

Prompted by your feedback, the revised paper discussed the goal of presenting Theorem 2. Our contribution is not the general analysis of biased SGD, but rather showing that even a restoration network not trained to be an ideal MMSE estimator can maintain stable convergence performance within our framework (as shown in Figure 2). This analysis is crucial as it provides theoretical grounding for the stability of ShaRP, especially in practical scenarios where the learned restoration model may be imperfect.

Although the authors show the performance of shaRP trained in an unsupervised manner, I don’t think the training loss used in the unsupervised case (e.g., the loss in Algorithm 3) corresponds to a restoration operator that approximates the conditional expectation $E[x|s,H]$. Consequently, the theoretical analysis does not explain the experimental results with a restoration operator trained in an unsupervised manner.

Thanks for pointing it out. We have included the full proof in the new Supplement A.3. As shown in the new Theorem 3, in expectation learning using the loss in Algorithm 3 is equivalent to learning using the fully-supervised L2 loss.

Page 1: “...Tweedie’s formula (Robbins, 1956; Efron, 2011) seemingly implies that Gaussian denoising alone might be sufficient for learning priors,...”: That is indeed true and this work does not disprove this fact.

Indeed, we do not seek to challenge the Tweedie formula, but rather emphasize that Gaussian-denoiser networks are suboptimal when used as priors to restore images degraded by other, non-Gaussian artifacts. Hence, the question is not whether Gaussian denoising can enable learning priors, but optimality of these priors when used to solve inverse problems.

Page 2: “...ShaRP provides a richer and more flexible representation of image priors…”: At this stage, it is rather vague as to what “richer and more flexible” means.

We revised that sentence for clarity: “The key benefit of ShaRP relative to prior work lies in its versatility, enabling seamless integration of a wide-range of restoration models trained on multiple degradation types…”

Page 2: “Unlike Gaussian denoisers, the restoration models in ShaRP can often be directly trained in a self-supervised manner”: Even Gaussian denoisers can be trained in an unsupervised manner using only noisy images (e.g., using a SURE loss).

Thanks for pointing this out. We revised that sentence for clarity: “Unlike Gaussian denoisers, the restoration models in ShaRP can often be directly trained without fully sampled measurement data.” Self-supervised Gaussian denoiser training methods, such as SURE, rely on noisy but fully-sampled data. We include an additional discussion highlighting the differences between self-supervised training for denoising and for restoration/reconstruction.

It might be good to put some of the math background in the intro, effectively shortening the material in the first three pages.

There is not much math that can be easily moved into the intro. However, we will make an effort to include more in the camera-ready version.

We thank the Reviewer for their time and feedback. Please see below for our point-by-point responses to your comments.

The paper seems limited in novelty and closely resembles [Hu et al., 2024c]. The main contribution is the training of a restoration prior across the same task with varying levels of ill-posedness. The authors should clearly explain how the proposed methodology differs from [Hu et al., 2024c].

The revised manuscript explicitly highlights differences between our work and Hu et al., [2024c]. Our work introduces a new theoretical framework and algorithm, resulting in significant performance gains as extensively detailed in the paper. The similarities are superficial and do not reflect the underlying novelty of our approach.

Summary of the new contributions of this work:

1. New regularizer: Our regularizer in eq. (6) is completely novel and different from that in [Hu et al. 2024c]. There is no analogous regularizer in the literature, as ours is the first one to provide a closed-form expression for a prior incorporating multiple restoration networks.

2. New algorithm: Algorithm 1 is a completely novel stochastic algorithm that is different from the deterministic algorithm in [Hu et al. 2024c]. To the best of our knowledge, no existing iterative reconstruction algorithm in the literature uses a degradation-restoration pair sampled at each iteration.

3. New theory: Our Theorem 1 is completely novel. It shows that our regularizer can be efficiently minimized using SGD. [Hu et al. 2024c] doesn’t have any analogous result. Training from undersampled measurements: Our ShaRP method doesn’t require fully sampled MRI measurements for training, which is not possible using the DRP method in [Hu et al. 2024c]. This flexibility is an essential for MRI and is a breakthrough not achieved by any existing PnP method in the literature.

4. Much better performance: We show that ShaRP outperforms DRP from [Hu et al. 2024c] across all experiments. It is on average 3.65 dB better than DRP on CS-MRI in Table 1 and 0.5 dB better than DRP on super-resolution in Table 3.

Thank you all for the additional feedback. One aspect of the review process that surprised us was the assessment regarding a perceived lack of novelty in our work. We respectfully disagree with this evaluation and emphasize that any resemblance to prior work is purely superficial. While we recognize that no paper is without limitations, our work introduces several new ideas, theories, and results that we believe warrant recognition as novel contributions.

1. More complex ensemble of denoisers: It is known that an ensemble of Gaussian denoisers trained at different noise levels does better than a single denoiser trained at one noise level. Our work shows that extending the “ensemble of denoisers” to include more complex (structured) noise types—due to each degradation operator—can further improve the performance. It is rather surprising that this non-trivial extension to a more complex “ensemble of denoisers” can still be used as an implicit prior. It is worth emphasizing that our framework can seamlessly incorporate both Gaussian denoisers and more complex structured denoisers into one closed-form implicit prior. This theoretical and conceptual novelty in our work deserves to be acknowledged.
2. Training from undersampled measurements: It is not possible to train Gaussian denoisers without fully-sampled measurements. Our framework provides an elegant solution by suggesting to train restoration networks on available undersampled measurements and using them as an implicit prior without retraining. This is a non-trivial contribution that deserves acknowledgement.
3. State-of-the-art performance: Our claims are backed up by impressive empirical performance on two separate inverse problems, MRI reconstruction and image super-resolution. This improvement due to the incorporation of non-Gaussian denoisers—or as we call them restoration operators—deserves to be acknowledged.