DreamClean: Restoring Clean Image Using Deep Diffusion Prior

Q: The DreamClean method can synthesize clean images given the corrupt images; however, the proposed method does not consider consistency between the corrupt images and the reconstructed clean images due to a lack of prior degraded models. The proposed method can fail some restored images by synthesizing unexpected clean images.

A: We agree that the consistency with corrupt images is important for IR problem. In this work, we also consider the consistency issue carefully but under much more challenging setting: the degradation model is unknown. Based on the fact that ODE sampling is approximately invertible, the inverted latents should contain desirable information about the input image. Hence, we leverage DDIM inversion to achieve approximate consistency. We admit that ODE inversion cannot ensure strict consistency with the input image without knowing the degradation model. It is an open problem and we are excited about the coming future that in-depth reseaches that can solve this problem thoroughly or mostly. Besides, orthogonal to previous methods, when the degradation model is known, DreamClean can also accomplish more faithful image restoration.

Q: The proposed method utilizes the pre-trained diffusion model on a particular large-scale dataset (i.e., ImageNet) to embed the corrupt images to the latent embeddings. Then, the method applies VPS to sample clean images during the sampling phase in diffusion models. Due to using the pre-trained model, the proposed method can generate unexpected images.

A: The concern that generating unexpected content because of the pre-trained model is reasonable. Like other generative methods for IR [1], our method also inherits the in-built stochasticity of diffusion models, which benefits diversity but with the risk of yielding undesirable results. As show in limitation (Sec. 5), diffusion models pretrained on ImageNet struggle to generate daily scenes such as altitude view of buildings. We figure out this problem clearly in limitation section (Sec. 5). It is an open problem that deserves further in-depth inverstigations.

Q: In section 3.3, which is quantitative experiments, the paper uses 100 NFEs rather than a configuration in the original paper for DDRM. Does it affect the quantitative comparisons because 100 NFEs may not be the configuration for DDRM?

A: We use $100$ NFEs for DDRM to keep the comparison in the context of the same or similar computation. In fact, as revealed in Table 1, DDRM with $100$ NFEs produces slightly better results than that with original $20$ NFEs.

Table 1: DDRM performace with different NFEs.

[1] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. ICLR 2023.

Q: For the proposed VPS, the latent variables fall into the high probability set when M is sufficiently large (Theorem 2.2). However, M is empirically set to 1 in this paper, and there is no detailed discussion about this important parameter.

A: $M=1$ works well because VPS is conducted for each timestep. This does not conflicts with Theorem 2.2 which states that when $M$ is sufficiently large, latents converges to a High Probability Set for certain timestep $t$. In other words, the correcting effect of VPS can be accumulated along the DDIM sampling trajectory. We have conducted noisy SR experiments on CelebA with different $M$ and the tendency of performance is plotted in Figure A2. The results suggests that increasing $M$ cannot significantly boost performance but with more compute. Therefore, we set $M=1$ for most cases.

Q: There are no visual comparisons to existing methods.

A: We follow the reviewer's advice to add visual comparisons with existing methods. Figures A7 and A8 presents visual comparisons of noisy SR and JPEG artifacts correction respectively. Figure A9 shows comparisons of $32$ SR with $\sigma=0.1$ using Stable Diffusion XL. It can be found that without knowing degradation model, DreamClean (ours in Figures A7, A8 and A9) can improve images' quality and produce visually pleasing results. When equipped with specific degradation, DreamClean (ours* in Figure A7) yields more faithful results.

Q: For the bad weather degradation, there are no quantitative measurements. The only evidence is a set of visual examples in Fig. 2.

A: We follow the reviewer's advise and include quantitative scores of image deraining on Rain100L. Table 1 shows the quantitative results. Baseline refers to original unprocess data. It can be found that without any specific training, DreamClean can effectively process complex rainy degradation. We also add more bad weather corrupted cases. Figures 2 and A12 present more bad weather degradation cases. Besides, Figure A1 presents more real-world bad weather cases.

Table 1: Quantitative results on Rain100L.

Q: The ablation study is only conducted on one type of degradation.

A: We follow the suggestion to perform ablation experiments on more degradation types. The below table shows that our VPS achieves the best score compared with other schemes. The reason is that VPS optimizes latents to High Probability Set, which conforms with the sampling dynamics of diffusion models. These results are consistent with noisy SR cases.

Table 2: Ablation study on more degradation types.

Q: The paper should discuss and compare with more recent blind image restoration works, e.g., GDP.

A: We follow the reviewer's suggestion to add comparison with GDP. For reviewer's convinience, we copy the quantitative results of noisy $4\times$ super-resolution on imagenet and CeleBA.

Table 3: Quantitative comparison with GDP.

Q: Is the proposed method sensitive to some parameters, e.g., step size $\gamma$ and strength $\tau$?

A: The effective range of $\gamma$ and $\tau$ is relatively loose. Empirically, we find that $\gamma \in [0.01, 0.1]$ and strength $\tau \in [300, 500]$ works well in most cases. To validate this, we conduct noisy SR experiments on CelebA with a range of $\gamma$ and strength $\tau$. As shown in Figures A3 and A4, our method works well when $\gamma$ and $\tau$ are in the above interval.

Thanks for your feedback! Below are our detailed responses.

Q: The correcting effect of VPS.

A: The form of VPS: $y_t^m = y_t^{m-1}+\eta_l\nabla\log p_t(y_t^{m-1})+\eta_g\epsilon_g^m$. We consider the degraded input consists of clean and corrupted components. We assume that it is the corrupted component that causes the low probability of latents and the amount of corrupted information remains constant. For arbitrary $t$, the first term $\eta_l\nabla\log p_t(y_t^{m-1})$ aims to increase its probability by removing low-probability corrupted component and the second term $\eta_g\epsilon_g^m$ adds a finely tuned amount of Gaussian noise to ensure that $y_t$ adheres to the formulation of diffusion models. It is worth noting that the addition of fresh Gaussian noise does not contribute to an increase in the level of corrupted information. This is because the amount of noise is carefully adjusted by $\eta_g$ to ensure that it can be totally removed after the sampling process of diffusion models. Therefore, in essence, VPS exchanges low-probability corrupted component for Gaussian noise, The amount of exchange is adjusted to match the dynamics of diffusion models. As this exchange occurs along the ODE trajectory and the total amount of corrupted information remains constant, the correcting effect accumulates gradually. To support this, please refer to visualization in Figure A5. Comparing latents of the interval $t\in [240, 400]$ of Figure a (without VPS) and Figure b (with $M=1$ VPS), we can observe the accumalated correcting effect. That is, at $t=400$, artifacts are still apparent after VPS. After accumulation along the sampling trajectory, at $t=240$, the artifacts are significantly reduced and the latent is like the clean image perturbed with Gaussian noise, which conforms the formulation of diffusion models. Additionally, we provide a theoretical analysis in Sec. A. 12 (Page 21).

Q: Include GDP for visual comparison.

A: We follow the advice to include GDP for visual comparison.

Q: The hallucination in bad weather.

A: The more hallucination in real-world bad weather cases can be explained that the degradation model is more complex and unknown. Therefore, the coupling of clean and corrupted component of the input image becomes more intricate in such scenarios. As we discuss in Limitation section (Secion 5, Page 9), DreamClean cannot guarantee strict faithfulness, i.e., without the knowledge of the degradatin model, DreamClean cannot provide complete separation between the clean and corrupted components of the input image. DreamClean provides a ''soft'' separation from the probabilistic prespective: the corrupted component should be evaluated as low probability using the metric of the generative prior learned from clean images. Therefore, the hallucination emerges because DreamClean resamples the low-probability component according to its generative prior learned from clean training images. The mentioned trade-off exists and can be implemented by the step size $\gamma$. In the limit that $\gamma=0$, VPS vanishes and DreamClean reproduces the input image using the DDIM inversion. As $\gamma$ increases, DreamClean regenerates the low-probability component.

Q: The performance needs to be tested on a broader range of real-world degradation data.

A: We follow the reviewer's suggestion to add more experiments on real-world degraded images. First, we evaluate quantitative performance of image denoising on the real-world SIDD dataset [2]. For comparison, we include two baselines: the scores of input noisy image and the classic denoising method BM3D [1]. Table 1 reveals the efficacy of DreamClean in real-world image denoising ($+8.26$ dB compared with Input). Besides, we also extend DreamClean to more real-world applications, ranging from real-world image deraining, desnowing and old photo restoration and real-world image denoising. As show in Figure A1, DreamClean can process complex real-world corrupted images. We have added related content in Sec. A.3.

Table 1: Quantitative results of real-world image denoising on SIDD.

Q: DreamClean should compare with recent training-free method GDP.

A:: We follow the reviewer's advice and add comparison with GDP. For reviewer's convinience, we copy the quantitative results of noisy SR on Imagenet and CeleBA.

Table 2: Quantitative comparison with GDP.

[1] Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. TIP 2007.

[2] A High-Quality Denoising Dataset for Smartphone Cameras. CVPR 2018.

Dear Reviewer aVXX,

The ICLR rebuttal discussion period is near the end. Does our response successfully address all your concerns? If so, would you please raise your rating? Your support is really important to us. If not, do please tell us your remained concerns. We are eager to discuss them with you.

Best wishes,

ICLR Submission 331 Authors.

Q: Why is ODE inversion needed rather than directly conduct VPS on the input image y?

A: ODE inversion indeed perturbs the data with noise. As revealed in Song et at [1], perturbing data can effectively mitigate the issue of inaccurate score estimation in low data density regions. With progressively decreasing noise, high-quality images can be generated. To further support this, experiments of noisy SR based on CelebA are conducted. Table 1 reveals that inversion provides more accurate score estimation, thereby boosting the final IR performance.

Table 1: Performance tendency with different ODE inversion strength.

Q: The authors should explain the reason that fresh noise is need in VPS. Why do higher-density latents perform inferior compared to latents with noise?

A: For certain timestep $t$, according to diffusion's formulation, latents should locate in the High Probability Set. As we state in Theorem 2.2, the amount of fresh noise is vital to ensure that the corrected latents is driven to the High Probability Set. Therefore, fresh noise is needed to conform the sampling dynamics of diffusion models, which produces high-quality images. Besides, as shown in Figure 8, without fresh noise ($\eta_g=0$), many details are discarded in the generated image.

[1] Generative modeling by estimating gradients of the data distribution. NeurIPS 2019.