Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration

W1. Direct evidence to support diffusion loss is effective

We are grateful for the reviewer's valuable suggestions on our validation experiments. Actually, our settings indeed followed such a validation way to evaluate if the proposed diffusion loss was effective. For example, in Table 1-3, the 'Vanilla' denotes the restoration model only trained with L1 loss on synthetic datasets; 'PixelDA' and 'CyCADA' (the mainstream domain adaptation methods) denote the restoration model trained with L1 loss on synthetic datasets and GAN loss on real-world datasets; 'Ours' denotes the restoration model trained with L1 loss and the proposed diffusion loss. To make our validation settings more intuitive, we have highlighted the training loss for each comparison method in Table 1 of the updated paper. Additionally, we have added a new experiment using perceptual loss to replace the diffusion loss. For your convenience, all quantitative results of the above validations are reported as follows.

From the experimental results, we have two main observations: (1) L1 loss and perceptual loss can effectively minimize average pixel and feature differences for image restoration. However, restoration models trained on synthetic images with these conventional loss functions still cannot escape from a significant drop in performance when applied to real-world domains. (2) GAN loss involves a discriminator distinguishing between real and synthetic images, pushing the generator to create more realistic outputs. However, they tend to be computationally demanding and unstable due to the need to train multiple networks and the complexity of the cycle consistency loss. By contrast, we propose a new noise-space solution that preserves low-level appearance across different domains within a compact and stable framework.

W2. Adding some visual results of other compared methods

Thanks for your kind suggestions. We have added the visual results of other compared methods into Figure A6-A8 in the updated paper. Please refer to the paper for more details.

W3. Can you provide training dynamics and complexity analysis of the diffusion model?

Absolutely. To clearly demonstrate the impact of the introduced diffusion loss during training, we have visualized the related metrics of training dynamics in Fig. A10 (left) of the updated paper. It is easy to find that the restoration model trained only with L1 loss on the synthetic dataset tends to overfit quickly and performs poorly on the real-world validation set. By contrast, the diffusion loss can effectively guide the restoration model to adapt to the real-world domain in a multi-step denoising manner, consistently improving the restoration performance on the real-world validation set.

Moreover, we agree the reviewer’s suggestion on validating the diffusion model with different complexities is an interesting exploration. We have tried this and shown the results in Fig. A10 (right) of the updated paper. To be specific, we classify the diffusion models into three types based on their complexities in each encoder layer: Diffusion-T: [32, 32, 64, 64], Diffusion-S: [32, 64, 128, 128], Diffusion-B: [64, 128, 256, 512] (exploited in this paper).

As we can observe, the real-world restoration performance gains further improvements as the complexity of the diffusion model increases, i.e., from Diffusion-T to Diffusion-B. We also provide a deep analysis of diffusion loss in restoration tasks when compared with MAR [1]: MAR models the per-token probability distribution using a small MLP as the diffusion model. It is trained jointly with the AR model to achieve efficient image generation. In particular, the tokens are small in size and represent high-level semantic features. The trained diffusion model has also been used during the reference sampling stage. By contrast, our diffusion model serves for the low-level image restoration problem. It directly adapts the restoration results at the dense and wide-scale pixel level, requiring accurate discriminations on the rich texture of images. The diffusion model is discarded after training, incurring no extra computational cost during restoration inference. Therefore, in the context of adapting the restoration model to the real-world domain, the diffusion model cannot be extremely simplified. The above discussions have also been added to the updated paper.

I have a follow-up question as the revised paper has some questions. What does $\mathcal{L}1$ in Table 1 refer to? (noise2noise approach? Ne2Ne approach? or what?) Also, is $\mathcal{L}_{diff}$ the same as Eq. 4 in your main manuscript?

W1. Clarification of the selected comparison methods

Previous domain adaptation works rarely focused on image restoration. To the best of our knowledge, our work represents the first attempt at addressing domain adaptation in the noise space for various image restoration tasks. Therefore, we mainly considered the classical domain adaptation methods and self-supervised methods as our comparison parts. Particularly, we selected MaskedD in the comparison since this work proposes masked training to enhance the generalization performance of denoising networks. The authors of MaskedD also claimed that "the proposed approach exhibits better generalization ability than other deep learning models and is directly applicable to real-world scenarios". It basically shares the same goal with our work. For more rigorous clarifications, we have highlighted the comparison settings and training data of different works in the "A1.2 Comparison Setting" of the Appendix.

Moreover, as mentioned in your following comments, our work contributes to a novel and general domain adaptation strategy for image restoration, which cannot be replaced by current self-supervised methods. The novelty of taming a diffusion model to achieve this goal has also been acknowledged by other reviewers. Thus, we further validated our method on the commonly used and SOTA restoration models, such as DnCNN, Restormer, and SwinIR, on the SIDD test set. The quantitative evaluations of these comparison methods are reported as follows. Particularly, Vanilla denotes training the model on the paired synthetic datasets, and Ours denotes training with the proposed domain adaptation strategy.

As we can observe, all restoration models trained on synthetic datasets failed to generalize well to the real-world dataset. By incorporating the proposed domain adaptation training strategy, the real-world performance of these models gains significant improvements, demonstrating the favorable generalization and scalability of our method. These new comparison experiments have been added to the updated paper.

W2. Considering more comparison methods such as PNGAN

Thanks for suggesting this comparison method. We agree that PNGAN is an excellent self-supervised work tailored for the image denoising task and we have included this work in our paper. However, we must honestly acknowledge that our experimental results do not surpass PNGAN, which can achieve a PSNR of 37.24dB when training the U-Net using its generated noises. This outcome may be anticipated, as our proposed method does not incorporate any prior knowledge of specific restoration tasks, aiming instead to provide a general solution for a wide range of restoration tasks. By contrast, self-supervised methods cannot perform well across different tasks and show limited generalization. Our experiments also demonstrated that by employing a more advanced restoration network, the proposed method can outperform some solid self-supervised learning methods such as C2N and AP-BSN (in Table 5).

We argue the difference between domain adaptation and self-supervised learning: Domain adaptation transfers knowledge from one domain to another with different data distributions, improving performance in new, unseen environments. Self-supervised learning, on the other hand, learns from unlabeled data by generating pseudo-labels or exploring the target distribution from the data itself. Both approaches reduce the reliance on large amounts of labeled data but address different challenges: domain adaptation focuses on bridging domain gaps, and self-supervised learning leverages data's inherent structure.

W1. Evaluation on fully unseen datasets

Thanks for helping us enrich the validation experiments. In the previous manuscript, we showed our visual results on fully unseen datasets such as DND and ‘Real-Internet’ in Fig. 8 and Fig. A6-A8. To further validate the generalization ability of the proposed framework, we have followed your suggestion and evaluated the performance on the PolyU dataset. The updated results are shown in the following table. All the evaluated methods are trained on the labeled synthetic dataset and real-world SIDD dataset (only degradation observation).

W2. An analysis of training time

As suggested, we provide a detailed comparison of training different methods. The evaluated results are reported in the following table. For a clear comparison, we show the total GPU hours (on NVIDIA A100 40G GPU) of training different denoising methods:

We also provide a detailed analysis here: Compared with previous domain adaptation and self-supervised methods, our method did not introduce significant burdens during training. As we can observe, our method is affordable and the whole training can be finished in one day using 4 GPUs. Additionally, as described in Section 3.2 and Appendix A4.1, our method can extend to the unpaired condition case by relaxing the diffusion’s input with the image from other clean datasets. Thus, the shortcut issue can be potentially eliminated since trivial solutions such as matching the pixel’s similarity between input and condition do not exist. Such an extension keeps the channel shuffling layer but is free to the residual swapping contrastive learning. As a result, the training cost can be further reduced as shown in "Ours_ex".

W3. Ablation studies on the sensitivity to hyperparameters

We agree the suggested experiments are necessary, and thus we provide the ablation studies on the sensitivity to hyperparameters as follows. In particular, we show the PSNR metric (dB) evaluated on the SIDD denoising test set. The experimental results demonstrate the moderate values for these two hyperparameters achieved the best performance, which means carefully balancing the roles of the restoration model and diffusion mode is crucial during their joint training.

W4. Analysis of the performance differences across different tasks

The improvement in image deraining (Table 2) shows that the improvement achieved by our method is apparent compared to that of the vanilla model, where +1.35 dB gain can be achieved in the PSNR metric. For the image deblurring task (Table 3), we acknowledge that +0.19 dB improvement is not as noticeable as other tasks; however, the performance still surpasses the classical domain adaptation methods and state-of-the-art self-supervised learning methods.

We also provided a discussion to analyze the different improvements across different restoration tasks as one of our limitations in Section 4.4. The related content has been presented here for your convenience: The natural mission of the diffusion model is to predict the noises mixed in the input, which is usually sampled from a high-frequency distribution. Diffusion models excel at capturing and modeling these small-scale variations due to their ability to learn fine-grained details through their denoising process. Thus, higher improvements can be observed in image denoising and deraining tasks, which typically involve high-frequency noises in images. By contrast, low-frequency noise in blurred images, which consists of smooth, gradual changes in intensity, can be more insensitive for diffusion models. This type of noise affects larger regions of the image and requires the model to correct broad, sweeping distortions rather than fine details. As a result, diffusion models may struggle to fully restore images with low-frequency noise compared to those with high-frequency noise. We leave it as one of the future works.

Thank you for your detailed and thoughtful response. While most of my initial concerns have been adequately addressed, the issue regarding the vague distinction between the training and inference stages remains only partially resolved.

In particular, the observed performance discrepancy on unseen datasets raises concerns. Unlike SIDD, which demonstrates significant improvement (+4–6 dB), the framework achieves only a modest gain on the on fully unseen Poly dataset (+1–2 dB). This observation strongly suggests that the adaptation process relies heavily on the noise distributions encountered during training, thereby limiting the practical applicability and generalizability of the proposed framework.

As a result, I have decided to retain my original score, as the effectiveness in broader, unseen scenarios appears constrained. I hope my feedback provides valuable guidance for improving this aspect in future work.

Best regards, Reviewer KUVz

Dear Reviewer kUVz,

We sincerely thank you once again for your insightful feedback. Perfectly generalizing a network's restoration capacity to fully unseen domains is indeed a meaningful and practical direction. However, given the significant domain gap, it remains an unsolved and open problem in the low-level vision community. To the best of our knowledge, no existing methods have achieved this ideal target.

Our work represents the first attempt to address domain adaptation specifically in the noise space for image restoration. We are also grateful that this contribution and its novelty have been recognized by you and other reviewers. Moreover, the performance comparison on the suggested fully unseen dataset highlights our noticeable performance gain over other methods.

Nevertheless, we honestly acknowledge that the performance improvement on the SIDD dataset is relatively higher than that on the PolyU dataset. This discrepancy is perhaps expected, as the PolyU dataset has lower noise density and a simpler distribution, which inherently limits the potential for significant improvement compared to the SIDD dataset, particularly in fully unseen settings.

In the final version of our paper, we will ensure that your invaluable suggestions are fully incorporated. If you have any additional questions or concerns, we would be more than happy to address them. Please feel free to leave a comment at any time. Thank you for your thoughtful review and constructive feedback!

Best regards, Authors.

W1. An explanation for "information leakage" involved in the training stage

We sincerely thank the reviewer for pointing out this interesting and insightful perspective of proactive "leakage". We would like to answer and discuss it from the following two aspects:

• Our motivation for this work is derived from an interesting observation: the prediction error of a conditional diffusion model relies on the quality of the conditions (as shown in Fig. 1 (a)). Therefore, guided by the back-propagated diffusion loss, the restoration network is optimized to provide "good" conditions to minimize the diffusion model’s noise prediction error, aiming for a clean image distribution. During this joint training, the synthetic GT serves as the denoising target in the diffusion model, which potentially offers realistic textures to help adapt the degraded real-world images into the clean distribution. In other words, the clean knowledge/information "leaked" by the diffusion’s input (in a multi-step denoising manner) plays an important role in bridging the gap between different domains.

• Generally, the ground truth images and those restored images by a restoration model, whether from synthetic or real-world domains, should reside within a common distribution of high-quality, clean images. However, the appearance of synthetic GT and restored real-world data is unrelated, leading the diffusion model to exploit a shortcut, overfitting its denoising capability by relying solely on the paired synthetic data. To this end, we further design crucial strategies (e.g., channel-shuffling layer and residual-swapping contrastive learning) to implicitly blur the boundaries between conditioned synthetic and real data and prevent the reliance of the model on easily distinguishable features. As a result, the proactive “leakage” from clean distribution to degraded images can effectively work during the whole training process, consistently improving the restoration performance on real-world images.

W2. Runtime for the proposed method

As suggested, we have updated the runtime for comparison methods as follows. All methods are tested on an NVIDIA A100 40G GPU.

Compared with previous methods, our work can achieve promising restoration performance on real-world datasets while showing favorable efficiency. This advantage stems from the flexibility of the proposed domain adaptation strategy, which only requires the conditional diffusion model during the training stage. The diffusion model is discarded after training, incurring no extra computational cost in restoration inference.

Q1. Could the authors show some failure cases?

Sure, the failure case has been added in the updated Appendix. Please refer to Fig. A9 for more details. In this case, we show our method failed to restore the images with challenging degraded distortions such as strong noises and out-of-distribution noises. These real-world degradations induce an extremely significant gap compared with the synthetic dataset, burdening the learning model to effectively adapt the restored results into the clean domain. Such a challenge also occurs on other comparison methods. It could be potentially addressed by incorporating more powerful diffusion models and we leave it as the future work.

W1. Show results where the background is not empty in de-raining images

Thanks for your suggestion. We have updated the de-raining results where the background is not empty. Besides, we also provided more de-raining results under various scenes in Fig. 8, Fig. A4 and Fig. A8. Please refer to the updated paper for more details.

Q1. Can the method be trained with multiple tasks simultaneously since it is technically (low-level) task agnostic?

We believe our work can be extended into such an application since the proposed framework offers a general and flexible adaptation strategy applicable beyond specific restoration tasks. It requires no prior knowledge of noise distribution or degradation models, and thus differs from self-supervised restoration methods tailored to one type of low-level task. We would like to leave this “all-in-one” network as our future work.

Dear Reviewer 5su7,

Thank you again for your review. We appreciate your recognition on our novelty and SOTA performance. We truly hope that our rebuttal can address your questions and concerns, such as showing results with more content and discussion on the "all-in-one" restoration model. As the discussion phase is nearing its end, we would be grateful to hear your feedback and wonder if you might still have any concerns we could address.

It would be appreciated if you could raise your score on our paper if we address your concerns. We thank you again for your effort in reviewing our paper.

Best Regards, Authors