Resfusion: Denoising Diffusion Probabilistic Models for Image Restoration Based on Prior Residual Noise

Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.

Since T' is smaller than the usually used T=1000, the sampling process of the proposed method seems to be faster than previous methods. But there are no comparisons on inference speed.

We demonstrate the comparison of different methods' inference speed on the ISTD dataset, LOL dataset and Raindrop dataset. When testing inference speed, all images are resized to $256\times256$, using single NVIDIA RTX A6000, with all configurations being identical to Appendix A.6:

We will certainly add the comparison of inference speed into Appendix A.6 of the revised paper.

What is the scheduling of \alpahs?

For all experiments, we used the truncated version of the Linear Schedule from reference [4] as the noise schedule for $\beta_t$ (or $\alpha_t$), which we refer to as the Truncated Linear Schedule. We provide a detailed implementation of the Truncated Linear Schedule in Appendix A.7.

The authors are suggested to add more experimental analysis about the error introduced by ignoring terms in Eq 6.

The ground truth of $x_{T'}$ is formulated as Eq. (7):

${x}_{T'}= (2\sqrt{\overline\alpha_{T'}}-1) x_{0}+(1-\sqrt{\overline\alpha_{T'}})\hat{x}_{0}+\sqrt{1-\overline\alpha_{T'}}\epsilon$

And the estimated acceleration point $\hat{x}_{T'}$ is formulated as Eq. (9) or Eq. (15):

$\hat{x}_{T'}= \sqrt{\overline\alpha_{T'}}\hat{x}_{0}+\sqrt{1-\overline\alpha_{T'}}\epsilon$

Since,

$R = \hat{x}_{0} - x_{0}$

The absolute value of the error can be derived as:

$||x_{T'} - \hat{x}_{T'}||$ $=||(2\sqrt{\overline\alpha_{T'}}-1) x_{0} + (1-2\sqrt{\overline\alpha_{T'}})\hat{x}_{0}||$ $=||(1-2\sqrt{\overline\alpha_{T'}})R||$

As shown in Figure 3 (a) of the Author Rebuttal PDF, the error $||(1-2\sqrt{\overline\alpha_{T'}})R||$ exponentially decreases with the increase of T. When T is relatively small, this error is not negligible. Fortunately, we can eliminate this error through the technique of Truncated Schedule when T is small (when T is large, Truncated Schedule is actually consistent with Original Schedule).

The core idea is that the diffusion steps after the acceleration point are not involved in the actual diffusion process. We provide a detailed implementation of the Truncated Linear Schedule in Appendix A.7. We provide a detailed code implementation of the Truncated Linear Schedule in Author Rebuttal Section 3.

As shown in Figure 3 (b) of the Author Rebuttal PDF, when $T'/T=5/12$, Truncated Schedule can effectively eliminate "residual shadows". The results of PSNR, SSIM, and LPIPS between Truncated Schedule and Original Schedule on ISTD dataset are provided:

We will certainly add the analysis of the error into the revised paper.

References

[1] Guo, Lanqing, et al. Shadowdiffusion: When degradation prior meets diffusion model for shadow removal. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[2] Wang, Tao, et al. Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 3. 2023.

[3] Özdenizci, Ozan, and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence 45.8 (2023): 10346-10357.

[4] Nichol, Alexander Quinn, and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. International conference on machine learning. PMLR, 2021.

Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.

How do you get/design Eq. (3)? I wonder if the term $(1-\sqrt{\alpha_t})R$ is manually designed or derived from a specific equation.

As shown in Figure 2 in the original paper, the resnoise-diffusion reverse process can be imagined as doing diffusion reverse process from $R+\epsilon$ to $x_{0}$ (as shown by the violet arrow).

$x_{t}$ can be represented as a weighted sum of $x_{0}$ and $R$ (in the forward process, we know the ground truth). At timestep $t$, to maintain consistency with DDPM [1], the coefficient of $x_{0}$ is determined as $\sqrt{\overline\alpha_t}$. Following the principles of similar triangles, the coefficient of $R$ at step $t$ is computed as $1-\sqrt{\overline{\alpha}_{t}}$, and we can derive Eq. (5).

Through the reparameterization technique, we can derive Eq. (3) (or Eq. (4)) and Eq. (5) from each other.

In Eq. (6), $\beta_t$ is not defined in the context.

In Eq. (12), how do you get the variance $\Sigma_{\theta}$ ?

For all constant hyperparameters, our definitions are completely consistent with the DDPM [1].

We will certainly add a detailed definition of the constant hyperparameters in the revised paper.

We provide a detailed definition (which is also declared in Author Rebuttal Section 1):

• $\beta_t = 1 - \alpha_t$
• $\overline{\alpha}_{t} = \prod_{s=1}^{t}\alpha _s$
• the $\Sigma_{\theta}$ is taken fixed as $\widetilde\beta_{t} = \frac{1-\overline{\alpha}_{t-1}}{1-\overline{\alpha}_{t}} \beta_t$

I can understand that Eq. (8) aims to ensure the coefficient of $x_{0}$ in Eq. (7) is close to 0. But how can you obtain Eq. (9) in which $\hat{x}_{0}$ is non-zero?

As defined in the first line of Section 2.1 in the original paper, $\hat{x}_{0}$ represents the input degraded image, which is obtainable during the resnoise-diffusion reverse process.

The overall derivation in Section 2.2 is unclear and confusing. Please make the connections between Equations more smooth.

We provide a detailed logical relationship for the equations in Section 2.2:

• To obtain a computable starting point, we introduce the smooth equivalence transformation technique, corresponding to Eq. (7) - Eq. (9).
• Because both the forward and backward processes only involve $t \le T'$, we unify the training and inference processes, corresponding to Eq. (10) - Eq. (14).
• We explained the working principle of Resfusion from the perspective of vector intersection, corresponding to Eq. (15).

We will certainly provide a more detailed logical relationship for the equations in Section 2.2 in the revised paper.

And how do you obtain Eq. (13)?

The derivation of $\mu_{\theta}$ corresponds to Eq. (19) - Eq. (23) in Appendix A.1. We provide a detailed explanation of the derivation process in Author Rebuttal Section 2.

We will certainly clarify the connection between Eq. (13) and Eq. (19) - Eq. (23) in the revised paper.

In experiments, it would be better to unify the evaluation metrics. For example, using PSNR, SSIM, LPIPS for all tasks and datasets.

We provide results in terms of PSNR, SSIM, and LPIPS for all tasks:

The proposed method seems to be sensitive to noise. Can you train the model for the denoising tasks?

The LOL-v2-real [8] dataset includes visual degradations such as decreased visibility, intensive noise, and biased color. As shown in Figure 2 of the Author Rebuttal PDF, Compared to Histogram Equalization, Resfusion can significantly reduce noise, while also achieving a better color offset, demonstrating strong denoising capabilities. We provide results in terms of PSNR, SSIM, and LPIPS on LOL-v2-real dataset:

We will certainly add experiments on the LOL-v2-real dataset into the revised paper.

References

[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.

[2] - [7] briefed due to word limit restrictions.

[8] Yang, Wenhan, et al. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing 30 (2021): 2072-2086.

Thank you very much for the patient response and constructive suggestions. Below we address specific questions and comments.

the derivation in Section 2.2 is still unclear

We provide a detailed logical relationship for the equations in Section 2.2 in our rebuttal:

• To obtain a computable starting point, we introduce the smooth equivalence transformation technique, corresponding to Eq. (7) - Eq. (9).
• Because both the forward and backward processes only involve $t \le T'$, we unify the training and inference processes, corresponding to Eq. (10) - Eq. (14).
• We explained the working principle of Resfusion from the perspective of vector intersection, corresponding to Eq. (15).

there is no answer to how to obtain Eq. (9)

Since

$\sqrt{\overline{\alpha}_{T'}}$

is closed to 0.5 as in Eq. (8),

$2\sqrt{\overline{\alpha}_{T'}}-1$

is closed to zero in Eq (7).

Then we can derive Eq. (9) from Eq. (7), where $\hat{x}_{0}$ represents the input degraded image.

I appreciate the authors' efforts in the rebuttal. For the notations and definitions in the paper, I want to note that although you use the same notations from other papers (e.g., DDPM), you still need to define/clarify them again in your draft so that the readers can understand them correctly (otherwise it would be confusing). I believe the main idea of this paper is similar to ResShift and RDDM. However, in the rebuttal, the derivation in Section 2.2 is still unclear and there is no answer to how to obtain Eq. (9). Therefore, I choose to maintain my original score i.e. * Borderline accept*.

Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.

ResShift also shifts toward the residual term. What are the advantages and disadvantages compared with ResShift?

We provide the differences between Resfusion and Resshift:

• Similar to RDDM [1], Resshift [2]'s forward process also adopts an accumulation strategy for the residual term and the noise term. Therefore, Resshift also requires the design of a complex noise schedule, which is formulated as equation (10) in reference [2]. Resfusion can directly use the existing noise schedule instead of redesigning the noise schedule.
• The reverse process of Resshift is inconsistent with DDPM [3]. The form of Resfusion's reverse inference process is consistent with the DDPM, leading to better generalization and interpretability.
• The prediction target of Resshift is $x_{0}$, while the prediction target of Resfusion is $res\epsilon$. Given that the essence of $res\epsilon$ is noise with an offset, and LDM models mainly predict noise, the loss function of Resfusion is extremely friendly to fine-tuning techniques such as Lora, which helps further scale up.
• In terms of the forward process, Resshift only performs shifting on the residual term. Resfusion not only shifts the residual term but also degrades the ground truth $x_{0}$, which ensures the image normalization.
• Resshift diffuses in the latent space, utilizing the powerful encoding capability of models like VQ-GAN. Resfusion, on the other hand, directly diffuses in the RGB space.
• Resshift only explores fixed degradations such as image super-resolution. Resfusion explores more complex degradations including shadow removal, low-light enhancement, and deraining.

We will certainly add the differences between Resfusion and Resshift into Appendix A.2 of the revised paper.

Visualization of the five inference steps is lacking.

We present the visualization results of the five sampling steps and use the pretrained model on the LOL dataset to directly infer images from the Internet (without ground truth). The visual results are stunning, as detailed in Figure 1 in the Author Rebuttal PDF.

We will certainly add the visualization in the revised paper.

What is the results with the complete inference steps (e.g., T=1000)?

We try $T'/T=100/272$ on the Raindrop dataset. All hyperparameters are consistent with the original paper. It is worth noting that increasing T will lead to an increase in training time (as the model needs to learn more scales of resnoise) and a decrease in inference speed. The results of PSNR, SSIM, and LPIPS are provided. Increasing T will result in a slight decrease in PSNR and SSIM (this may be due to the training set and test set not being completely i.i.d.), but it will yield better visual perception (LPIPS).

A branch out: Will reducing T in traditional diffusion-based models increase PSNR and SSIM?

We try WeatherDiff$_{64}$ [4] on the Raindrop dataset, the answer is definitely NO. The results of PSNR, SSIM, and LPIPS are provided.

In discussion, what is the model size? Can a 7.7M model achieve such good results in Figure 7?

As mentioned in discussion and detailed in Appendix A.4, for image generation on the CIFAR10 ($32\times32$) dataset, we utilize the same U-net structure as DDIM [5]. The parameter size of the denoising backbone is 35.72M.

References

[1] Liu, Jiawei, et al. Residual denoising diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

[2] Yue, Zongsheng, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. Advances in Neural Information Processing Systems, 2024.

[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.

[4] Özdenizci, Ozan, and Robert Legenstein. Restoring vision in adverse weather conditions with patch-based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelligence 45.8 (2023): 10346-10357.

[5] Song, Jiaming, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.

Thank you very much for the recognition of our work and rebuttal. We are delighted to have addressed your concerns. We would greatly appreciate it if you could consider raising the score.

Thank you for the detailed review and thoughtful feedback. Below we address specific questions and comments.

Several notations are used without being defined. For example, the definitions of $\alpha_t$, $\beta_t$, $\overline{\alpha}_{t}$ are missing.

For all constant hyperparameters, our definitions are completely consistent with the reference [1].

We will certainly add a detailed definition of the constant hyperparameters in the revised paper.

We provide a detailed definition (which is also declared in Author Rebuttal Section 1):

• $\beta_t = 1 - \alpha_t$
• $\overline{\alpha}_{t} = \prod_{s=1}^{t}\alpha _s$
• the $\Sigma_{\theta}$ is taken fixed as $\widetilde\beta_{t} = \frac{1-\overline{\alpha}_{t-1}}{1-\overline{\alpha}_{t}} \beta_t$

Some mathematical derivations are not detailed enough. For example, the proof in Appendix A.1 is not sufficiently detailed.

By simply performing a change of variable ($x_0 \rightarrow x_0-R$, $x_t \rightarrow x_t-R$, $x_{t-1} \rightarrow x_{t-1}-R$), the derivation of Eq. (18) is identical in form to (71)-(84) in the reference [2], where line 6-7 of Eq. (18) corresponds to (73).

We will certainly add a detailed derivation of Eq. (18) in the revised paper.

Another example is derivation of $\mu_{\theta}$ in Equation (13), which is missing.

The derivation of $\mu_{\theta}$ corresponds to Eq. (19) - Eq. (23) in Appendix A.1. We provide a detailed explanation of the derivation process in Author Rebuttal Section 2.

We will certainly clarify the connection between Eq. (13) and Eq. (19) - Eq. (23) in the revised paper.

To provide a more comprehensive background, it might be beneficial to cite [3] which introduced a general framework for leveraging generative diffusion for solving image restoration problems where the observed image is contaminated by linear degradation and additive white Gaussian noise.

SNIPS [3] combines annealed Langevin dynamics and Newton's method to arrive at a posterior sampling algorithm, recovering the clean images from the white Gaussian noise. As a pioneer in exploring the generative diffusion processes to solve the general linear inverse problems ($y=Hx+z$), SNIPS has made remarkable contributions to image restoration. By incorporating the residual term into the diffusion forward process, Resfusion recovers the clean images directly from the noisy degraded images. Also, Resfusion explores more complex scenarios where the degradation $H$ and the noise level are unknown, such as shadow removal, low-light enhancement, and deraining, achieving competitive performance.

We will certainly introduce SNIPS [3] in the background of the revised paper.

References

[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.

[2] Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022.

[3] Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. Advances in neural information processing Systems 34:21757-21769, 2021.

Dear reviewers, as we approach the final two days, please take a moment to review the author's responses and join the discussion. Thank you!

We sincerely appreciate Area Chair Ydph. Your responsible feedback has greatly inspired us. We would also like to express our gratitude to all the reviewers for their professional review and recognition of our work.