Stochastic Deep Restoration Priors for Imaging Inverse Problems

We thank the reviewer for their valuable feedback.

1. For concerns in Methods And Evaluation Criteria:

Prompted by your comment, we ran additional experiments that we will include in the supplementary material of the paper. The new setting considers non-Cartesian sampling for CS-MRI, using a 2D Gaussian sampling mask. Importantly, the restoration network trained for the 8× uniform Cartesian sampling was applied directly without retraining or adaptation. Our results, summarized below, provide additional evidence of ShaRP's robustness to diverse sampling patterns, highlighting its strong generalization ability.

It would certainly be interesting to explore additional natural distortions in SISR tasks, such as motion blur, in future work. We will mention this revised manuscript.

2. For concerns in Theoretical Claims:

Thanks for your suggestion. We will include an expanded discussion about the two technical assumptions. Lipschitz continuity is a standard and widely-used assumption in optimization, needed for establishing convergence rates of gradient-based algorithms (see, for example, Section 1.2.2 in [1]). It is satisfied by a broad class of objective functions, including the least-squares data-fidelity term used in our experiments, as well as neural networks (e.g., Appendix B in [2]). Boundedness is a mild assumption, since it is always true for images that have bounded pixel values [0, 255].

[1] Nesterov, Introductory Lectures on Convex Optimization, 2004 [2] Hurault et al. Gradient step denoiser for convergent plug-and-play. ICLR, 2022.

3. ..., but no ablation study quantifies how ShaRP’s gains scale with ensemble size b.

Section E.2 of the supplementary material presents an ablation study quantifying performance gains with varying ensemble size (b). Computationally, ShaRP remains efficient because it uses only one restoration model from the ensemble at each inference step, thus having comparable computational complexity to single-model methods. The revised manuscript will ensure that this study is clearly referenced in the main manuscript for visibility.

4. SISR uses only two Gaussian blurs, which is not enough.

Thanks for your suggestion. Prompted by your comment, we ran an additional SISR experiment using a new Gaussian kernel (σ = 1.0). As shown in the table below, we can achieve consistently good performance as the other two kernels in the manuscript. Note how ShaRP still provides excellent performance, suggesting that it would work with more Gaussian blurs.

5. Perceptual quality requires human evaluation.

Prompted by your comment, we introduced FID and LPIPS as perceptual metrics. Please refer to the rebuttal comment (7) to Reviewer DuT7 for the table. The revised manuscript will mention that true perceptual quality requires human evaluation.

6. The two papers shown below handle similar tasks: [1] Osmosis, ECCV, 2024. [2] DreamClean, ICLR, 2024.

We will cite these papers in the revised manuscript.

7 Technical terms (e.g., "restoration operator") are inconsistently defined.

We will review the use of “restoration operator” across to make sure it is consistent. Restoration operator in our paper refers to an operator that computes MMSE solution of eq. (3).

8. Supplement should provide experiments with other degradation types.

We will include results reported in our response to your Comment 1 to the supplementary material.

9. How does ShaRP’s performance/complexity scale with ensemble size?

Please refer to our response to your Comment 2.

10. Could ShaRP adapt to settings without even partially sampled data?

We are not entirely sure what the reviewer means by “without even partially sampled data.” If the reviewer is referring to the setting where there are no measurements at all, we can still run our method, but it was not conceived for this setting. Indeed, it would be very interesting to extend our methodology to “unconditional generation” or “sampling”, i.e., generation of images from the prior specified using a set of restoration operators. This will be a great direction for future work.

11 How did inexact restoration operators impact convergence in practice? Does the bias term remain stable?

See Figure 7 in Section C.3, which shows stable convergence even with inexact (self-supervised) MMSE restoration operators. Overall, we did not observe any stability issues with ShaRP.

We thank the reviewer for their valuable feedback.

1. It is unclear what ShaRP is after - is it a better MAP estimate? or perhaps an MMSE one? Clearly, this is not a posterior sampler. This question is critical, ..., as they they tend to be somewhat blurry.

This is a great point. ShaRP is not designed to produce classical MAP or MMSE estimates, nor is it a posterior sampler. Instead, it optimizes a novel objective function (Eq. (2) using h(x) in Eq. (6)) that balances data fidelity with an implicit regularization learned from an ensemble of priors. While the resulting reconstructions may exhibit characteristics similar to MMSE solutions (e.g., in terms of PSNR and visual smoothness), this is a consequence of the learned objective rather than an explicit design goal. We acknowledge that exploring sampling-based extensions could potentially enhance perceptual performance, and we will include this as a direction for future work.

2. In any case, a directly trained MMSE solver should be included ...

Thank you for the suggestion. For 4× CS-MRI, we added E2E-VarNet as a new baseline, which is a dedicated MMSE solver with data-fidelity constraint that achieves higher PSNR but is task-specific (i.e., [*] needs to be retrained for each task), effectively representing an upper bound on MSE performance.

3. Also, as this work compares various different methods ... perceptual quality evaluation.

See our answer to Comment 4 below.

4. Several comments above suggest that the comparisons are not complete and not fair: FID/KID results are missing - does ShaRP aim for high perceptual quality? If not, say so, but then, what is it aiming for? MMSE restoration is missing.

We clarify that ShaRP primarily aims for high reconstruction accuracy rather than explicitly targeting perceptual quality. Nevertheless, we agree perceptual quality metrics provide valuable insights (in particular regarding the Perception Distortion tradeoff [3]). Hence, we have included FID alongside PSNR, SSIM, and LPIPS to offer a balanced assessment:

[3] Y. Blau., and T. Michaeli. "The perception-distortion tradeoff." CVPR, 2018.

5. A reference to RED-Diff is very much missing.

We will cite RED-Diff in the revised manuscript.

6. The paper is hard to follow in Section 3 - the order of the presentation is flawed ...

This is valuable feedback. We will implement the following changes: We will begin by clearly defining all necessary functions and terms, with a specific definition of h(). Following this, we will introduce Algorithm 1. We believe this revised structure will significantly improve the clarity and flow of Section 3.

7. A clear definition of the obje3ctive or ShaRP is missing ...

The objective function minimized by ShaRP is the composite function consisting of a data fidelity term g and a novel regularizer h specified in eq. (6). See also our response to Comment 1.

8. In equation 3, why not have sigma be random as well? ...

While ShaRP generates a random noise at each iteration, it does so using fixed sigma. Using random sigma is an interesting idea that we could explore in the future.

9. Equation 6 is very hard to follow, and esspecially so since you do not show the chosen h() beforehand. It would be helpfull to show how the actual h() is generalized to lead to (6).

We will revise the paper to better explain Equation (6).

We thank the reviewer for their valuable feedback.

Response to concerns in Claims and Evidence

1. Generalization over configurations: We wanted to highlight that the supplementary material contains evidence supporting the robustness of ShaRP across diverse inverse problems without retraining, particularly under self-supervised conditions from incomplete measurements. Specifically: As detailed in Section C, we conducted experiments in the CS-MRI setting evaluating ShaRP's performance under a range of challenging conditions, including: (1) Different undersampling rates (4x, 6x), (2) Varied mask types (Cartesian uniform, Cartesian random, and 2D Gaussian), (3) Multiple noise levels (0.005, 0.1, 0.15). These results demonstrate ShaRP's ability to handle diverse inverse problems without retraining, even with self-supervised learning from incomplete data.

2. Convergence Stability and Robustness: Section C.3 of the supplementary material presents an analysis of ShaRP's convergence stability and robustness. We evaluated this using both supervised and self-supervised MMSE restoration priors, providing empirical support for our theoretical guarantees.

Prompted by the reviewer comment, the revised paper will include: (1) A summary table that consolidates the key ablation results across different undersampling rates, mask types, and noise levels. This will provide a clearer and more accessible overview. (2) Add a brief discussion in the main text highlighting these empirical findings and explicitly linking them to the claim of robustness without retraining. We believe these additions will strengthen our paper by highlighting extensive evidence provided in the paper.

Response to concerns in Theoretical Claims

Prompted by the reviewer the revised paper will better highlight the following:

1. Lipschitz continuity and boundedness are needed only for Proposition 1. They are not needed for our main results—Theorem 1 and Theorem 2.
2. Lipschitz continuity is a standard and widely-used assumption in optimization for establishing convergence rates of gradient-based algorithms (see, for example, Section 1.2.2 in [1]). It is satisfied by a broad class of objective functions, including the least-squares data-fidelity term used in our experiments. The smoothness of MMSE restoration operators is well-known and has been extensively discussed in literature (see, for example, [2]).
3. Boundedness is a mild assumption, since it is always true for images that have bounded pixel values [0, 255].
4. Figure 7 in Section C.3 shows empirically convergence of our method in a real-world setting.

[1] Nesterov, Introductory Lectures on Convex Optimization, 2004

[2] Gribonval and Machart, Reconciling “priors” & “priors” without prejudice?, NeurIPS 2013.

Response to concerns in Experimental Designs Or Analyses

1. Prompted by your comment, we will include a table summarizing all the relevant information about the experiment to guarantee reproducible results. We will additionally release our code upon acceptance, which should further simplify reproducibility.

2. Section E.2 of the supplementary material shows clear performance gains as ensemble size increases.

Respond to concerns in Essential References Not Discussed

Thank you for pointing to relevant works such as "Deep Image Prior," "Noise2Void," and "Self2Self." We will cite and discuss these works in our revised manuscript.

Response to concerns in Other Strengths And Weaknesses

We appreciate the reviewer's assessment of our paper's originality and contributions. Following your recommendations, the revised manuscript will include summary tables pointing to the relevant results and will improve the discussion on the assumptions (see response to your Comment 2).

Response to concerns in Other Comments Or Suggestions

The revised manuscript will explicitly mention that the computational cost of running ShaRP is comparable to those of single-model approaches. This is due to the stochastic nature of our algorithm that uses a $single$ restoration operator in each iteration.

Response to concerns For Authors

1. The smoothness of h associated with the MMSE restoration operators is well-known (see for example [1] or Appendix B of [2]). The smoothness can also be seen in the derivation presented in Appendix A of our paper. This is a direct consequence of the smoothness of the p(s | H) in eq. (5), since it is a Gaussian convolved with the prior p(x). Note also that since our restoration operator is implemented as a neural network with smooth activation functions like eLU, it will be inherently Lipschitz continuous [2].
2. (1) The detailed results of ablation studies isolating the effect of the ensemble are available in Section E.2, which is in the supplementary material. (2) There is no computational overhead due to our approach since only one model from the ensemble is utilized for each restoration instance. We will explicitly state this in the revised manuscript.

We thank the reviewer for feedback and thoughtful comments on our work.

1. The paper differentiates itself from related self-supervised methods by stating "Ambient DMs seek to sample from px using DMs trained directly on undersampled measurements. Thus, during inference Ambient DMs assume access to the image prior px, while ShaRP only assumes access to the ensemble of likelihoods of multiple degraded observations." However, because p_x was learned directly from undersampled measurements, the assumptions on Ambient GAN and related methods don't seem any more restrictive than those on the current method (which I would argue has also implicitly learned p_x).

Indeed, both Ambient DMs and ShaRP indeed implicitly learn about p(x) and we will revise that section of the paper to make this point clear.

2. We introduce a novel regularization concept for inverse problems that encourages solutions that produce degraded versions closely resembling real degraded images." This seems conceptually similar to the equivariant imaging concept. Can the authors comment on the relationship between the two works. Is there a relationship between Theorem 2 and equivariant imaging?

This is a very interesting perspective. Our approach can indeed be seen as finding a fixed point that exhibits equivariance under multiple degradations. The revised paper will include this nice interpretation suggested by the reviewer.

3. May be worth discussing the following paper: Bansal, Arpit, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. "Cold diffusion: Inverting arbitrary image transforms without noise." Advances in Neural Information Processing Systems 36 (2023): 41259-41282.

We will cite and discuss Cold Diffusion in the revised version. Both Cold Diffusion and Ambient Diffusion (mentioned above) point toward a promising direction for extending our method to sampling, which we will highlight as a potential avenue for future work.

4. It might be better to rewrite (1) and (2) with complex-valued forward models, given the focus on MRI.

We will fix this per your suggestion.

5. At least in the context of an ensemble of blur-then-deblur operations, ShaRP can be interpreted as masking then restoring a portion of k-space: ShaRP could be viewed as an ensemble of masked (denoising) autoencoders.

Indeed, when considering an ensemble of blur-then-deblur operations, ShaRP can be seen as masking and subsequently restoring portions of k-space—akin to the mechanism of masked (denoising) autoencoders. In this interpretation, MMSE restoration networks effectively reconstruct the missing null-space signals. We will incorporate this connection into the discussion section of our revised manuscript.

6. A subscript A in $\nabla(g)$ in Algorithm 1 (i.e., $\nabla_A(g)$ ) might be useful to remind readers where the forward model comes into play in the reconstruction algorithm. (Or just some additional annotation that g(x)) is the data fidelity term.)

We will modify the notation accordingly.

7. What is s' in (12)? Are the losses ell_sup and ell_self wrt \bar{x}? The notation in this section could use more annotation.

In Eq. (12), s’ refers to an independently subsampled measurement, defined as s’ = P’M with P’ denoting a separate sampling pattern. We will revise the notation and add clarifying annotations in the revised version.

8. In the MRI restoration task, the trained restoration algorithms are solving problems very similar to the target application; both are solving subsampled MRI, just with different masks. Is the proposed method still effective when the restoration algorithms are solving a very different task? E.g., can I solve compressive MRI by regularizing with an algorithm trained to perform image deblurring or inpainting?

Section E.1 of our paper explores the scenario suggested by the reviewer by applying a pre-trained super-resolution (SR) model to the compressive sensing MRI problem. Despite the task mismatch, the SR prior still outperformed the Gaussian denoiser prior. We will edit the manuscript to ensure that it is clear this experiment was included.