Masked Pre-training Enables Universal Zero-shot Denoiser

$**Q1: More comparison with zero-shot modifications of blind spot methods**$

$**A1:**$ Thank you for your thorough consideration. We have revised AP-BSN, MM-BSN, and PUCA (as shown in the overall rebuttal to all reviewers), and the results are listed in $*Table 1*$ in provided PDF. Here are some Implementation details below:

We followed the original settings of these methods. In each iteration, we cropped 8 same-size patches from a noisy image to form a batch for training. Every 10 iterations, we performed inference on the full image to obtain denoised images. These denoised images were then combined using the same ensemble strategy as our method to ensure fairness.

These methods can effectively denoise spatially correlated noisy images. However, using only one noisy image for training can lead to overfitting and produce artifacts, as shown in $*Fig. 5*$ of the PDF.

$**Q2: More discussion about masking ratio**$

$**A2:**$ Thank you for your suggestion; it is very insightful. We are still exploring this issue. However, we are concerned that using a large masking ratio during inference might degrade synthetic noise removal performance. Here is a summary of the impact of different masking ratios $p$ on denoising:

Small $p$: Risk of learning noise patterns from real noisy images.

Large $p$: Loss of detail in the denoised image.

The difference in masking ratios for various noise types is mainly because real noise has strong spatial correlation, necessitating a larger masking ratio to avoid learning the noise distribution (see more analysis in $**A3**$).

Choosing the masking ratio is crucial. When noise is spatially uncorrelated (e.g., Gaussian, Poisson, S&P noise), a consistent $p$=30 works well across all cases. However, the spatial correlation of real noise complicates this issue. Many blind-spot networks (e.g. AP-BSN) designed for self-supervised denoising also aim to address this problem.

$**Q3: Main reason for different$p$**$

$**A3:**$ This primarily depends on the spatial correlation of the noise in the image. Synthetic noise is spatially uncorrelated, with noise signals are uncorrelated in neighbor positions. In contrast, real noise, after passing through a series of ISP processes, has a much more complex distribution, resembling blurred spots rather than independent points (refer to $*Fig. 2*$ in PDF). For synthetic noise, choosing a small masking ratio can help quickly recover more details in the image. However, for real noise, with a small masking ratio, the model can fit the noise distribution based on neighboring pixel values. In this case, a larger masking ratio can mitigate the impact of noise. One paper [Ref1] also discusses using different dropout ratios for different types of images.

$**Q4: Minor points and typos**$

$**A4:**$ Thanks very much for your corrections.

1. Regarding Fig. 2 and Fig. 6 in main text, we have provided enlarged views, as seen in $*Fig. 1 and Fig. 4*$ in PDF.

2. Yes, we have addressed the spacing issue on line 89.

3. "2.3" on line 248 refers to Section 2.3, which we have clarified in the main text.

4. Due to formatting constraints, we shortened the caption. "Defaults in gray" means that the gray background in the table indicates the default settings used in our work for comparison with other methods.

$**Ref:**$

Ref1: Self2Self+: Single-Image Denoising with Self-Supervised Learning and Image Quality Assessment Loss, Arxiv'23

$**Q1: Similarities and differences compared to MetaDIP and DGP**$

$**A1:**$ Thank you for your pointing out. Our method shares some similarities with MetaDIP and DGP. MetaDIP learns denoising by obtaining initial weights beneficial for downstream tasks, while our method uses masked training on natural images to enhance downstream denoising. However, MetaDIP and DGP seem to require known degradation models, which may not be applicable for some noise types. In contrast, our method learns to recover denoised images from masked noisy ones, without relying on models tailored to specific degradation types. This eliminates the need for degradation modeling, making it adaptable to unknown noise types, real noise, and various image types.

$**Q2: Comparison fairness issues caused by ensemble**$

$**A2:**$ Thanks for your attention. We acknowledge that ensemble technique are existing methods. Our focus is on demonstrating the advantages of masked pre-training priors.

To ensure fair comparison, we explained this in the experimental section of the main text. We used EMA ensemble for DIP and FasterDIP, as detailed in $*lines 170-173*$ in main text. Results for other comparison methods (N2V and N2S) with EMA ensemble are shown in $*Supplementary Materials G.2 (lines 600-608, Tables 12 and 13)*$. Our “faster” version performs not the best in some settings, but the 1000-step version consistently leads.

We chose the EMA version for other methods because our experiments showed that EMA yields the best results compared to other ensemble methods (like averaging), as detailed in $Table 5 of the main text.

$**Q3: Comparison with other DIP methods**$

$**A3:**$ We considered the work you mentioned (DIP-SURE) when selecting comparison methods. They designed denoising solutions for Gaussian and Poisson noise to achieve high-quality denoising. However, their method requires additional noise variance as input, which could lead to unfair comparisons. Additionally, their method seems to be specific to certain types of noise, and their official code only supports Gaussian and Poisson denoising. This is why we did not compare with their method initially.

We included results for DIP-SURE using EMA ensemble, reporting both peak performance and final performance. For a fair comparison, we should compare our method with the final performance, as peak PSNR is not known without ground truth. Please refer to $*Table 1*$ in PDF for a comparison between our method and DIP-SURE.

For real datasets (SIDD, PolyU, and FMD), we computed the variance from the difference between noisy and clean images to obtain denoised results. Their approach cannot remove real-world noise well, and some artifacts exists (refer to $*Fig.5*$ in provided PDF).

$**Q4: Comparison with other Diffusion methods**$

$**A4:**$ Existing SOTA diffusion models, like DDNM, recover degraded images by decomposing into the null-space and adjusting the noise scheduler for denoising. Although diffusion models are Gaussian denoisers and can handle Gaussian noise well with proper adjustments (see PDF $*Table 1*$), they also require known noise variance in advance. Additionally, when set for denoising, the diffusion model relies solely on the noise scheduler, effectively becoming a Gaussian denoiser. This limitation may prevent the model from fully removing more complex real-world noise (see $*Fig. 5*$ in PDF).

Dear reviewer,

thanks for your review. Please look at the rebuttal and give the authors some feedback whether they could address your concerns.

Best regards Your AC.

$**Q1: Lack of essence of our method**$

$**A1:**$ Thank you for your pointing out. The core of our method lies in the inherent denoising capability of a model pre-trained with masked natural images. This motivation is demonstrated in main text. The pixel-level random masking we employ can be viewed as a form of noise, disrupting the image structure, and the model is trained to restore these structures. Due to training on a large set of natural images, the limited model parameters are unable to precisely fit all image distributions and instead tend to prioritize learning the main features of the images while discarding noise that varies across different images, thus acquiring a certain denoising capability. Similar masked training strategies are used in many self-supervised denoising tasks [Ref4, Ref5, Ref6 in author rebuttal], where models are trained on a large set of noisy images to learn the corresponding clean images. We further extend this to a zero-shot version, combining it with a large amount of easily accessible natural images to enhance the performance of zero-shot methods.

$**Q2: What model learns from pre-training and how it is applied to zeroshot denoising**$

$**A2:**$ We address your concerns from three aspects:

$*What does the model learn from pre-training?*$

The model learns to restore randomly masked image content from a large set of natural images. This restoration process is somewhat noise-resistant. In short, a model pre-trained on a large dataset tends to recover denoised image content, functioning as a kind of denoising autoencoder (see $*Fig.1*$ in the PDF for motivation).

$*How is it applied to zero-shot denoising?*$

The pre-trained weights are iteratively optimized on a noisy image through random masking, and predicted pixels are integrated using an exponential moving average, resulting in a denoised image. The learned denoising representation provides a better initial weight and helps avoid over-fitting to the noisy image.

$*Analysis of why pre-trained features helped zero-shot denoising.*$

We analyzed the impact of pre-trained features on zero-shot denoising at the hidden layer level (see $*Fig.3*$ in the PDF). Features extracted with pre-trained weights significantly differ from those extracted from scratch (Baseline, i.e., the usual zero-shot denoising approach). The pre-trained model restores the complete image, with more distinct features between layers, whereas the baseline model's features are less differentiated between layers, tending to only restore the masked parts, leading to local optima.

$**Q3: Why a model does not learn denoising can denoise and generalize to unnatural images**$

$**A3:**$ As per $**A1**$, we believe that the model pre-trained with masked images acquires a certain robustness to noise, enabling it to perform denoising. Regarding its applicability to non-natural images, the use of pixel-level random masking allows the model to restore masked content based on the unmasked areas. Since the unmasked pixels are spatially distributed evenly, the model tends to focus more on local, low-level features such as texture and color rather than high-level semantic features. These low-level features share some commonalities across all types of images, allowing the model to be applicable to significantly different types of images, such as medical images and natural images.

$**Q4: Motivation of our approach**$

$**A4:**$ Thank you for pointing out, our motivation comes from the blind spot network in self supervised denoising, which learns to reconstruct noisy images cropped by blind-spots (much resemble pixel-wise random masking) to denoise on noisy datasets that do not include ground truth. This is a widely explored field, and we further expand it with using natural images to obtain a better zero shot denoising algorithm.

$**Q5: Why only discuss denoising**$

$**A5:**$ Thank you for your suggestion. Our motivation stems from findings in the self-supervised denoising field, so this paper focuses solely on denoising. As per $**A1**$, the trained model tends to acquire smooth representations beneficial for denoising. Since our current model uses a small number of parameters for efficient denoising, it might struggle to generate sufficient details in other tasks such as deblurring or super-resolution. However, we believe that with further improvements, this method has great potential in other types of degradation tasks.

$**Q6: Answer to Questions**$

$**A6:**$

1. During the inference phase, we generate predictions using random masks and iterative filling, with the final denoised image comes from the exponential moving average of multiple inferences for each pixel. The only difference is that we use weights pre-trained on natural images with random masking as the initial weights, which speeds up inference and reduces the risk of over-fitting.

2. Regarding what the model has learned and why it denoises, we believe that masked pre-training on large datasets causes the model to focus on the main features of images (relatively lower-frequency, easier-to-recover features), which gives the model robustness to noise and enables it to perform denoising.

3. we have analyzed the differences in features between pre-trained and from-scratch models during inference to demonstrate the advantages of pre-training over starting from scratch.

Dear reviewer,

thanks for your review. Please look at the rebuttal and give the authors some feedback whether they could address your concerns.

Best regards Your AC.

I have read the author's response, as well as the comments and discussions with other reviewers. The author has partially addressed my concerns. However, I still think that the presentation of the paper is lacking at this stage. I will improve my score. However, since I cannot see the revised paper, I cannot judge whether the final presentation meets the requirements of NeurIPS.

Thank you for your suggestions. We studied the three works you provided carefully. The first is a score-based denoising algorithm, the second is an improvement on blind-spot networks (PUCA), and the third is a diffusion-based image restoration method (DDPG). The first two seem to be unsupervised denoising approaches based on datasets, which learn to recover denoised images unsupervisely from a dataset containing only noisy images.

For the first paper, we could not find their code on GitHub. Reproducing this paper within a week is challenging, so we did not compare our method with theirs.

For the second paper, we provide results for its modified zero-shot version PUCA* and the original PUCA in the table below. The zero-shot method uses an ensemble approach to ensure a fair comparison.

For the third paper, we provide results (DDPG) in the table below. This diffusion method requires the noise variance as known information. We use the variance calculated from the difference between the noisy and clean images to ensure the best performance for this method (though this might risk ground truth leakage).

We have also compared several other self-supervised and diffusion methods. If interested, please see $*Table 1*$ in PDF. For the visual results on SIDD, see $*Fig. 5*$ in PDF.

$**Analysis:**$ Blind-spot methods like PUCA handle strong spatially correlated noise well in real-world scenarios. However, in its zero-shot version, it may risk over-fitting due to a lack of sufficient training data.

Diffusion methods like DDPG can handle various types of degradation and are inherently robust to noise due to their training on Gaussian noise. However, they struggle to generalize to real-world noise scenarios.