Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

We are grateful for the reviewer's affirmation of the effectiveness of our method, especially the outstanding performance in music separation and partial generation tasks. Below are our responses.

1. DDNM+ numbers for noisy Deblurring tasks

We are very grateful to the reviewer for pointing out this issue. We used the official code of DDNM in the noisy linear inverse problem. We double-checked the official code and confirmed that it indeed calls the DDNM+ algorithm. However, we found that the implementation of Gaussian Deblurring in the official code seems to be incomplete, which is the reason for the abnormal results in our paper.

We re-implement the part of Gaussian Deblurring and test it on both ImageNet and CelebA. The corrected results are shown in Table 5 of the attached PDF, where DDNM+ achieved reasonable results. We also checked Super-resolution and Inpainting and confirmed that their implementations are correct. We will correct the experimental results in our paper and annotate that we used the DDNM+ algorithm for noisy tasks. We would like to express our gratitude once again to the reviewer for helping us correct this error.

2. ''What it means to not fully leverage diffusion models''

Our observation is that diffusion models not only model the distribution of clean data $p(x_0)$, but they also implicitly model the distribution of noisy data with different variances of Gaussian noise $p(x_t)$ within the diffusion process. This means that compared to other types of data priors, diffusion models can provide more usable information for solving inverse problems with Gaussian noisy observations. Therefore, if solved within a conventional MAP framework, only the clean data prior modeled by the diffusion model would be utilized while the noisy data prior within the diffusion model is neglected.

In contrast, we introduce an auxiliary variable to transform the noisy observation into an equivalent noise-free observation of the noisy sample, thereby utilizing both the clean data prior and the noise data prior. The process of recovering clean data from the noisy auxiliary variable can be regarded as a denoising process, hence our method actually utilizes both the data prior modeled by the diffusion model and its denoising capability.

3. More comparative algorithms and related works

Thanks for the reviewer's suggestion. We supplement a comparison of ProjDiff with $\\Pi$GDM, together with ReSample, DiffPIR, and DMPS on the CelebA dataset. The results are shown in Table 1 & 2 of the attached PDF. ProjDiff still demonstrates competitive performance.

For $\\Pi$GDM, we carefully conducted hyperparameter searches for each task using the first eight images with the average PSNR as the metric. Except for Noisy Gaussian Deblur task, we avoided using other algorithms to initialize $\\Pi$GDM to ensure a fair comparison. Specifically, we first use the forward transition to map the degraded image to the noise level corresponding to the 500th step, and then use 100 steps of $\\Pi$GDM to solve the inverse problem. We find that this approach is much better than starting directly from white noise (i.e. 1000th step). For Noisy Gaussian Deblur task, $\\Pi$GDM without initialization from other algorithms struggled to achieve satisfactory results. Therefore, we use DDNM+ to obtain the sample at the 500th step and then switch to $\\Pi$GDM to complete the solution.

We notice that $\\Pi$GDM does not always outperform the performance of our reproduced DDRM. This may be because we report the performance of DDRM using 100 steps, which is much better than the default parameters of DDRM (20 steps).

We will further refine the explanation of related work according to the reviewer's suggestions.

4. DPS on CelebA

Thanks for the reviewer's suggestion. We have re-adjusted the learning rate hyperparameter $\\eta$ for DPS on the CelebA dataset task by task. Using the average PSNR of the first eight images as the criterion (consistent with the method we used to adjust the hyperparameter of ProjDiff), we carefully scanned the learning rate hyperparameter with a precision of 0.1. The latest results are shown in Table 1&2 of the attached PDF (denoted as DPS-s). We will update these results in the paper and include an explanation.

5. ProjDiff for latent diffusion

Yes, we believe ProjDiff can be applied in latent diffusion. The main challenge in solving inverse problems with latent diffusion priors is the high nonlinearity of the observation equation introduced by the neural encoder-decoder. ProjDiff has the capability to handle nonlinear observations. Firstly, for noise-free observations, ProjDiff can approximate the projection operation by taking $x_0$ as the initial point and minimizing $||y-\\mathcal{A}(\\mathcal{D}(x))||^2$ (where $\\mathcal{D}$ represents the neural Decoder). Secondly, for noisy observations, ProjDiff also needs to set an equivalent noise level, which can be left as an adjustable hyperparameter. Our supplemental experiments in Table 7 of the attached PDF have verified that ProjDiff is robust to some perturbations in the noise variance (i.e. equivalent noise level), making manual adjustment of the equivalent noise level feasible. We believe this is a promising direction for future work.

We would like to express our gratitude once again for the reviewer's suggestions for our work. Should there be any further questions, we welcome continued discussion.

I would like to thank the time and effort authors put into their rebuttal. I appreciate that they have incorporated all the feedback into providing a fairer comparison (fixing DDNM+, tuning DPS step size, etc.) against baseline methods especially in a short window of time.

Most of my concerns are alleviated and my questions are answered. I've also read the comments of other reviewers (and authors' response to them). I believe the results are more convincing with the updated numbers and the proposed changes. Therefore, I'm happily updating my score to $6$ and increasing soundness to $3$.

We are grateful for the reviewer's recognition of the innovation and effectiveness of our work. Below are our responses.

1. Reorganizing the theoretical analysis and the proofs of the lemmas.

Thanks for the reviewer's suggestion. We will move Lemma 1 and 2 ahead of the proofs of the Propositions, and we will present the proofs of the two Propositions in a clear and independent manner. The proofs of Lemma 1 and 2 are based on Bayes' theorem and mathematical induction, and we will supplement the proofs of this part.

2. Abstract includes some concerning statements.

Thanks for the reviewer's feedback. We will revise the abstract accordingly.

The first sentence will be revised as:

''Previous works have endeavored to integrate diffusion priors into the maximum a posteriori estimation (MAP) framework and design optimization methods to solve the inverse problem.''

The second sentence will be revised as:

''... by introducing an auxiliary optimization variable that represents a 'noisy' sample at an equivalent denoising step. The projection gradient descent method is efficiently utilized to solve the corresponding optimization problem by truncating the gradient through the $\\mu$-predictor.''

We will further refine the abstract to make it more fluent and reflective of our approach.

3. The authors could state how their method is different from the literature and how it contributes to the literature.

Compared to previous works that apply diffusion in the MAP framework for solving inverse problems, our core observation is that diffusion models not only model the distribution of clean data but also implicitly model the distribution of noisy data. This allows us to leverage both the clean data prior and the noisy data prior to solve noisy inverse problems (in other words, both the prior modeled by the diffusion model and its denoising capabilities). Thanks for the reviewer's suggestions, and we will further improve the expression of paragraph 3 and paragraph 4.

4. The reduced computational cost

Thanks for the reviewer's comments. We did not intend to present the reduction of complexity as a contribution of this paper. Line 44 will be revised to:

''We obtain a more practical approximation of the stochastic gradient of the objective through gradient truncation.''

Additionally, we have added an experiment comparing the use of our gradient approximation with not using it, as shown in Table 6 of the attached PDF. It can be observed that using the approximation results in a certain loss of performance, but the efficiency is nearly tripled. Considering that the method using this approximation has already achieved satisfactory performance, we accept a certain performance loss in exchange for efficiency.

5. Line 19.

Thanks for the reviewer's suggestions. Line 19 will be revised to

''Their remarkable ability to capture data priors provides promising avenues for solving inverse problems [6]...''

6. The weak observations problems.

We refer to inverse problems where the constraints provided by the observation are relatively loose as ''the weak observation problem''. For instance, the partial generation task in this paper aims to generate tracks for other instruments based on certain tracks (e.g., generating tracks of drums, bass, and guitar given a track of piano). This problem is highly flexible as a composer can create various different tracks for other instruments from the same piano track while ensuring harmony. This is a case belonging to the ''the weak observation problems''. In contrast, tasks such as super-resolution, inpainting, and deblurring in image restoration are cases where the observation strongly constrains the original data. For example, given a low-resolution image, its corresponding high-resolution image is almost unique, with lower degrees of freedom, and thus it does not belong to the ''weak observation problems''.

7. Regarding nonlinear measurements

The algorithm for nonlinear measurements is almost identical, except that the projection operations in equations (19) and (22) are replaced with the projection operations corresponding to nonlinear measurements, or by minimizing $||y-\\mathcal{A}(x))||^2$ from the initial point to approximate the projection operation.

For general nonlinear measurements, equation (8) cannot be strictly written. However, we can still apply a similar idea to handle nonlinear measurements, which is to simply find an equivalent noise level to deal with noisy observations. This equivalent noise level can either be calculated using rules or retained as a hyperparameter to adjust. Our Phase Retrieval experiments and HDR experiments have verified the effectiveness of our method.

8. The hyperparameters

All of the details of our hyperparameters are provided in Appendix E and F. Regarding the sensitivity of hyperparameters, we supplement a new experiment in Table 7 of the attached PDF. It can be observed that our algorithm exhibits robustness to the step size and the noise variance.

We would like to express our gratitude once again for the reviewer's suggestions for our work. Should there be any further questions, we welcome continued discussion.

We are grateful for the reviewer's recognition of the innovation, effectiveness, and practicality of our work. Below are our responses to the reviewer's questions.

1. Clarification on the motivation and comparison to the established MAP approach.

The core motivation of this paper stems from the fact that diffusion models not only model the distribution of clean data $p(x_0)$, but also simultaneously model the distribution of noisy data with varying variances of additive Gaussian noise $p(x_t)$. When observing a degraded data with Gaussian noise, the noisy data distribution can bring more usable information for solving the inverse problem. Therefore, compared to using only the prior information modeled by the diffusion model for $x_0$, we choose to use the prior information of both $x_0$ and $x_{t_a}$. Our algorithm can be viewed as two steps: first, recover the noisy sample $x_{t_a}$, and then use the consistency constraints of $x_{t_a}$ and the prior of $x_0$ to recover $x_0$ (which can be viewed as a denoising process). In other words, we utilize both the prior modeled by the diffusion model and its denoising capabilities.

The conventional approach of including the logarithm of the forward model in the optimization task is also feasible, as seen in [24, 28] in the main paper, and we have compared with it in our experiments (RED-diff). The improvement of our algorithm compared to these methods can be expected because our method utilizes more information modeled by the diffusion model, i.e., it uses both the prior and denoising capabilities (or both the clean data prior and the noisy data prior). We believe this is the essential difference between our algorithm and the methods that include the log of the forward model and $\\log p(x_0)$ modeled by diffusion models in the MAP framework.

2. Further explanation on SVD-based decoupling.

Here we provide additional clarification regarding the description in section 2.3. Firstly, since $V$ is orthogonal, we acknowledge that knowing the prior of $x_0$ is equivalent to knowing the prior of $\\overline x_0=V^T x_0$. Secondly, since the covariance matrix of both the forward transition and a single-step backward transition of the diffusion model is the diagonal matrix, each coordinate can be processed individually. Equation (7) presents the observation equation in a coordinate-wise form, where for each coordinate index $i$, the $i$-th coordinate of the noisy observation $\\overline y_i$ can be considered as a noise-free observation of the $i$-th coordinate of the sample $\\overline x_{t_i}$ at step $t_i$. In other words, each coordinate of the noisy observation (after spectral decomposition) can be considered as a noise-free observation of the coordinate at a certain step in the diffusion process.

Regarding the reviewer's concern about a convolution with a blur kernel, it is not about viewing the noisy blurred image as a step in the diffusion process, but rather, after SVD, each coordinate is treated as the noise-free observation of the coordinate corresponding to a certain step in the diffusion model. We hope the above explanation could describe our method more clearly. Thanks for the reviewer's suggestions, and we will make further modifications and explanations regarding the SVD part of the paper.

3. Regarding the limitations.

Yes, the limitation of this method is that it can only handle noise-free inverse problems or inverse problems with additive Gaussian noise. Addressing other types of noise, such as Poisson noise or multiplicative noise, is a promising direction for further research. In practical applications, noise-free inverse problems or inverse problems with additive Gaussian noise are widespread, hence there is a series of work in the field of solving inverse problems with diffusion models that assumes Gaussian noise (including but not limited to [18, 20, 21, 22, 26, 27] in the main paper). Therefore, our work holds practical significance.

We would like to express our gratitude once again for the reviewer's suggestions for our work. Should there be any further questions, we welcome continued discussion.

We appreciate the reviewer's suggestions, and here are our responses.

1. ''The assumption for the measurement noise is doubtable.''

We respectfully disagree. Firstly, in practical applications, noise-free inverse problems or inverse problems with Gaussian noise are ubiquitous, and the standard deviation of the Gaussian noise can be estimated using the method mentioned by the reviewer or retained as an adjustable hyperparameter. Secondly, a series of works (including but not limited to [18, 20, 21, 22, 26, 27] in the main paper) that use diffusion models to solve the inverse problems assume additive Gaussian noise and known variance (DMPS mentioned by the reviewer also assumes additive Gaussian noise with known variance, and the DiffPIR mentioned by the reviewer also assumes the variance is known). We believe that these works, as well as ours, have practical value. Lastly, as shown in the ablation study in response to point 4, ProjDiff is robust to some perturbation of the standard deviation. Therefore, our work holds practical significance.

2. ''The approximation after Proposition 2 is not convincing.''

We respectfully disagree. Firstly, this approximation is acceptable when $t$ is small, as the $\\mu$-predictor generally learns well to recover the clean image from a slightly noisy image. Secondly, for large $t$, in the case of VP diffusion, note that the coefficient $\\overline\\alpha_t$ in front of $x_0$ or $x_{t_a}$ is small, thus the gradients corresponding to $x_0$ and $x_{t_a}$ are also small. For VE diffusion, similarly, the dominant term in $\\mu$ will also be the noise term when $t$ is large, making it insensitive to the input $x_0$. Our experimental results have demonstrated the effectiveness of this approximation. Furthermore, we supplement a new experiment that uses the stochastic gradient without this approximation, as shown in Table 6 of the attached PDF (denoted as ProjDiff-FG). It can be observed that using this approximation results in a certain loss of performance, but the efficiency is nearly tripled. Considering that applying this approximation has already achieved satisfactory performance, we are willing to accept a certain performance loss in exchange for efficiency.

3. ''The proposed method is too 'delicate'.''

We respectfully disagree. Once again, assuming knowledge of the variance of Gaussian noise is a default setting in many works within the field. Secondly, as we explain in Appendix F, we fix $\\eta_2=1.0$, thus only $\\eta_1$ needs to be tuned (as well as the equivalent noise level $\\overline \\alpha_{t_a}$ if we leave it as a hyperparameter to be tuned). We conduct hyperparameter tuning on the first eight images for each task. Given that the complexity of ProjDiff is not too high, tuning hyperparameters is not time-consuming. In the ablation study in response to point 4, we also demonstrate that ProjDiff is robust to some perturbation of the step size and the standard deviation (i.e. the equivalent noise level). To the best of our knowledge, many related works have at least one hyperparameter that needs to be adjusted, including the algorithms mentioned by the reviewer.

4. Ablation studies for hyper-parameters

We supplement the ablation study concerning $\\eta_1$ and ${\\sigma}_0$ in Table 7 of the attached PDF. ProjDiff exhibits robustness to some perturbation of the hyperparameters of the step size and the noise standard deviation.

5. ''The extension to nonlinear inverse problems is not convincing''.

The reviewer has misunderstood our approach. One of the core aspects of ProjDiff is to project a sample $x_0$ onto the manifold corresponding to $y=\\mathcal{A}(x)$. When $\\mathcal{A}$ is nonlinear whose projection operator is unavailable, one can instead find a solution that is close to $x_0$ and close to the observation manifold by solving $\\min_x||y-\\mathcal{A}(x)||^2$ starting from $x_0$. This solution can serve as an approximation for the projection operation corresponding to a nonlinear observation equation. We also supplement an experiment to validate the effectiveness of this method in the Phase Retrieval task (ProjDiff-GD in Table 4 of the attached PDF).

6. ''The experiment settings for phase retrieval are doubtable.''

We respectfully disagree. Our experimental settings for phase retrieval are correct, fair, and reasonable. All of our experimental settings are detailed in Appendix E. We generate one sample per image for each algorithm. The good performance of ProjDiff under this setup further demonstrates its robustness. We also supplement the results of allowing ProjDiff and DPS to generate 4 samples for each image, as shown in Table 3 of the PDF (denoted as ProjDiff-4trials and DPS-4trials). Both DPS and ProjDiff show performance improvements, with ProjDiff in particular exhibiting better performance, reaching a PSNR of 41.58 in the noise-free case. The performance of HIO, ER, and OSS is also presented in Table 3 and the global response, where we report the results with both one repetition and four repetitions.

Note that DPS does not achieve the PSNR of 30 as mentioned by the reviewer. To dispel further doubts about the correctness of our reproduction, we independently run the official code of DPS four times without any modifications (i.e., noise variance set to 0.05, with oversampling consistent with our experiments) and take the best result per sample, obtaining a PSNR of 17.45, indicating that our reproduction is also correct.

7. Comparison with the latest SOTA.

We supplement comparisons of ProjDiff with ReSample, DiffPIR, DMPS, and $\\Pi$GDM on the CelebA dataset, as shown in Table 1 & 2 in the attached PDF. The hyperparameters are tuned task by task for fairness. ReSample is designed for Latent Diffusion, thus in our experiments, we consider the encoder-decoder as identity mappings. ProjDiff remains highly competitive when compared with these algorithms.