DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency

Thank you for your time reviewing our manuscript. We are glad to hear that our method is “a valuable contribution” and “has practical relevance for the solution of inverse problems”. Please find below our response.

• Re supervised training baseline: Thank you for your suggestion. We added a state-of-the-art supervised image restoration baseline, SwinIR, to our experiments on the FFHQ dataset. We observe that SwinIR achieves excellent performance in terms of distortion metrics which is closely matched by DiracDiffusion optimized for distortion. However, as expected, DiracDiffusion (and other diffusion-based solvers typically) outperforms supervised non-diffusion restoration methods in terms of perceptual metrics by large margins. We are in the process of running the same experiments on the ImageNet dataset as well, which we are going to add to the final version of the manuscript. However, we expect to see the same trends as in the case of FFHQ.

• Re train-test degradation mismatch: This is a great question. We added a new robustness study with respect to forward model perturbations in test time (see Appendix H). In particular, we perturb the blur kernel width in test time in Gaussian deblurring. We find that, as in other supervised methods, reconstruction performance degrades when the degradation model is significantly changed. However, we observe that performance is almost unchanged under blur kernel standard deviation reductions of up to 20%, which is a significant perturbation. Moreover, the reviewer can find experiments on the noise robustness of DiracDiffusion in Appendix G and H (Figure 8), where we perturb the measurement noise level in test time. We observe that DiracDiffusion is robust with respect to noise level perturbations. We hypothesize that this is due to the fact that our model is trained to denoise on a range of noise variances due to the similarity of our objective to denoising score-matching.

We hope that we have adequately addressed the reviewer’s concerns and we are open for further discussion if needed. We would greatly appreciate it if the reviewer considered updating their rating in case all issues have been addressed, given the reviewer’s generally favorable opinion on our paper.

Thank you for reviewing our paper and we are glad to hear that the reviewer found it interesting and novel. Please find below our response to the reviewer’s concerns.

• Re needs retraining: This is a great point, in fact a limitation of our work is that it needs to be trained for a specific reconstruction task. However, the extra cost in training time pays off greatly in faster inference (we only need 5-20 NFEs for optimal performance vs 1000 in DPS), improved reconstruction fidelity and additional flexibility in trading off distortion for perceptual quality. We would argue that in settings where the degradation model is fixed, our model has clear benefits over other diffusion-based solvers.

• Re other image reconstruction tasks: Thank you for raising this point. In fact, our technique can be used for any image corruption, where the functional form of the degradation is known (superresolution, JPEG compression artifact removal etc) and can be interpolated between the most severe degradation (t=1) and identity (t=0). We hope that the provided experiments sufficiently support the utility of our proposed framework. It would be interesting to further tailor DiracDiffusion for specific popular inverse problems, however this effort is more geared towards future work.

We hope that the reviewer’s concerns have been adequately addressed and we are open to addressing any further questions or concerns. We would greatly appreciate it if the reviewer could update their rating if all issues have been addressed, given that the reviewer assigned generally good scores to our paper.

We thank the reviewer for their effort reviewing our work. We are glad that the reviewer thinks that the problem is “interesting and relevant” and that the paper is “well written and the theoretical presentation is sound and clear”. Please find below our reply to your concerns.

Re Question 1: Thank you for bringing up these relevant works.

• Re Daras et al. (2022): We build upon the work in Daras et al. (2022) and propose a novel framework specifically tailored for inverse problem solving and extending the formulation to general corruptions (they consider linear corruptions only). In particular, Soft Diffusion is defined as a solely generative framework, and thus it does not cover sampling from a posterior distribution given noisy measurements. Therefore, the considerations around data consistency, which is the main focus of our work, does not arise in their setting.

• Re Delbracio and Milanfar (2023): In fact, we did compare our method with InDI (see Appendix G for a discussion and experimental results ). Overall, we found that InDI and DiracDiffusion provide similar perceptual reconstruction quality (we achieve slightly better FID, InDI has slightly better LPIPS). However, we have found that DiracDiffusion is more robust with respect to noise level perturbations in test time. We hypothesize that this is due to the fact that our model is trained to denoise on a range of noise variances due to the similarity of our objective to denoising score-matching. However, InDI reconstructs samples from convex combinations between noisy and clean images, and thus the model has not seen any other noise levels but the one used for training. We expect similar observations for Bridge methods as well. We would also like to highlight that these techniques work under a different framework from ours, where the forward model is unknown and thus their focus is not on data consistency (in fact it cannot be evaluated without knowing the forward process).

Re Question 2: Thank you for this suggestion. As we mentioned above, we provided a comparison on FFHQ with InDI which is a trained diffusion model-based solver. However, at the time of writing this paper the source code and checkpoints have not been publicly released and thus we had to rely on our own reproduction of the method with computationally heavy training. We are unfortunately unaware of diffusion solvers with fast fine-tuning for adaptation to various inverse problems, but we are happy to compare if the reviewer has some suggestions. Moreover, we added a strong supervised image restoration baseline (SwinIR) to our main result in the newer version of the manuscript that the reviewer might find useful. Based on the feedback, we would be open to moving the discussion on InDI to the main paper in the final version of the manuscript.

Please let us know if you have any further questions or suggestions. We hope that we managed to address the reviewer’s major concerns, in which case we would really appreciate it if the reviewer could update their rating given the generally favorable review.

Thank you for taking your time reviewing our paper. We are glad that the reviewer thinks that “the proposed method is sound” and that the “experimental results well verified the effectiveness of the proposed method”. Please find below our response to your concerns.

• Re Weakness 1: As the reviewer points out, DiracDiffusion is a solver for inverse problems with known forward model. This is a clear distinction from general image restoration techniques that rely only on clean-noisy paired data for training without knowing the corruption process. However, in this general scenario there is no notion of data consistency: given a corrupted input and the reconstructed image we are unable to evaluate how well the reconstruction explains the input (without a ground truth). In stark contrast, our work focuses on inverse problems with known forward models, a common case in real-world problems where the physical process producing the observations is known (known camera blur, microscopy with known probe function, medical imaging problems such as CT or MRI etc). In this setting, a crucial requirement is data consistency: the reconstruction has to be consistent with the noisy measurement as dictated by the forward model. In fact, ensuring this consistency while producing exceptional image fidelity is the main goal of our paper. We would like to also highlight that other diffusion-based solvers overwhelmingly assume a known forward model as well ([1-3]). To address the reviewer’s concern we updated the main text to better emphasize this assumption.

• Re Weakness 2: Our early stopping method is indeed straightforward and intuitive, yet very effective. Furthermore, the early stopping technique is an interesting consequence of our proposed Stochastic Diffusion Process framework, and not the sole contribution of our work.

• Re Weakness 3: Thank you for pointing this out, we added the specific form of the degradation process in Appendix F. We emphasize that $\mathcal{A}_t(\cdot)$ is designed a priori based on the known forward model and its time-dependence (scheduling) is determined via the degradation scheduling algorithm in Algorithm 2 in the appendix.

• Re Weakness 4: We point out that when the degradation process is linear ($\mathcal{A}_t(\cdot)$ is a linear mapping), reconstructing $\mathcal{A}_t(x_0)$ is not equivalent to reconstructing $x_0$ as the linear mapping is typically not invertible (as is the case in deblurring or inpainting). In fact, as we highlight in the paper, predicting $\mathcal{A}_t(x_0)$ from the noisy and degraded input (denoising only) is an easier problem than completely reconstructing $x_0$ in one step (denoising and reconstruction/inversion).

Thank you for catching the typo, we fixed it in the new version. We are happy to discuss further if there are additional concerns and would like to ask the reviewer to consider raising their score in case we addressed all major issues.

[1] Kawar, Bahjat, et al. "Denoising diffusion restoration models." Advances in Neural Information Processing Systems 35 (2022): 23593-23606.

[2] Chung, Hyungjin, et al. "Diffusion posterior sampling for general noisy inverse problems." arXiv preprint arXiv:2209.14687 (2022).

[3] Song, Yang, et al. "Solving inverse problems in medical imaging with score-based generative models." arXiv preprint arXiv:2111.08005 (2021).