Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum

We sincerely thank Reviewer 5arj for the positive review and helpful suggestions. Your comments have helped us better clarify important methodological details and highlight the robustness of our approach. Below, we would like to address each of your questions and concerns:

Question 1: How important will the temperature of the curriculum be?

Thank you for the thoughtful question. Our curriculum does not explicitly involve a temperature parameter. Instead, we adopt a discrete two-phase schedule, transitioning from denoised (easier) to noisy (harder) data. We experimented with gradually increasing the proportion of noisy samples during training but found that this neither accelerated convergence nor improved final performance.

Currently, the softmax temperatures used in the DINOv2 joint embedding loss are set to the default schedule for both training stages. The teacher temperature follows a warm-up phase and then remains constant. The student temperature is held constant. We are aware that, in the knowledge distillation literature [1], temperature can be dynamically learned to maximize the distillation loss, thereby increasing learning difficulty and forming an effective curriculum. While in our paper the learning difficulty is primarily controlled by the presence of noise, we think exploring the incorporation of dynamic temperature into our research is also highly worthwhile. We appreciate your insight that this is an interesting direction for future exploration.

[1] Curriculum temperature for knowledge distillation Li et al., AAAI 2023  
 

Comment 1 on the weakness: The noise is random, and may vary across many runs, something that the experiment at larger scales (data size) avoids, raising a question over scaling.

Thank you for raising this concern, and we apologize for the lack of clarity. We would like to clarify that the noise does not vary across runs in any of our experiments. We generate the noisy dataset once prior to training and do not apply noise on-the-fly during data loading. This setup is intended to closely simulate real-world scenarios where one starts with a fixed noisy dataset, aligning with realistic settings. We will make sure to explicitly clarify this detail in Section 4.1 (Noise Addition) of our revised paper.  
 

Comment 2 on the limitation: The noise added is …added. Multiplicative or mixed source often plagues data.

We fully agree with your valid concern that multiplicative or mixed noises often plagues data. We would like to clarify that many noises tested in this paper, like shot noise and speckle noise, are not additive. In particular, speckle noise is a multiplicative noise, as shown in its formulation in Appendix A.5 equation (9).

To address the concern regarding mixed noise sources, we evaluated our method on Poisson-Gaussian noise, which is one of the most common mixed noises in imaging. The Poisson-Gaussian noise is formulated as $x = \frac{\operatorname{Poisson}(\tilde{x} \cdot \lambda)}{\lambda} + \mathcal{N}(0, \left(\frac{\sigma}{255}\right)^2)$, where $\tilde{x}$ is the clean image. We set $\lambda = 3$ and $\sigma = 100$, using the default 100 epochs to train the N2N denoiser, and 200 epochs to train DINOv2 ViT-S on ImageNet-100. The linear probing accuracies are shown in the table below:

Comparison of linear probing accuracies on Poisson-Gaussian mixed noise with 200 epochs fixed training budget

The results are fully consistent with Figure 1 and Table 5 in the paper, where DINOv2 w/ NC significantly outperforms the DINOv2 baseline, and DINOv2 w/ NCT closely matches N2N + DINOv2. This demonstrates our methods are effective under mixed noise conditions and generalize to more complex scenarios. We will ensure to include these results in our revised paper to highlight the robustness of the methods under mixed noises.  
 

We appreciate your thoughtful feedback and hope we have addressed all of your questions and concerns. We remain engaged and willing to provide further clarification during the ongoing discussion.

We sincerely thank Reviewer 1LLb for the positive feedback, and for raising a series of very insightful questions. Below, we would like to give detailed responses to each of your suggestions and questions.

Question 1: Performance of DINOv2 trained on clean images evaluated on the noisy test set?

Thanks for raising the necessity of this baseline and we fully agree that it is crucial for establishing this work’s significance. To address this, we conduct further evaluation to supplement Table 2 with the results for both ImageNet-1k and ImageNet-100, as shown below. We observe that significant drops in accuracy occur (>20 points for Gaussian-100, >40 points for Gaussian-255) when evaluating the clean-pretrained DINOv2 on the noisy test set. This demonstrates SSL models trained exclusively with clean data cannot generalize to noisy images in downstream tasks, thus it is important to research training methods that produce noise-robust models, which is the main goal of our paper. We will update Table 2 and plan to include the additional results for Table 3 in our revised paper.

Question 2: Show results when training on mixed noisy + denoised images

We understand the reviewer’s valid suspicion that incorporating denoised images could enhance the training set’s expressiveness. In practice, we found mixing clean (denoised) and noisy images during training often leads to degraded performance. This is primarily due to reduced representation alignment between noisy and denoised samples, which hinders the model’s ability to learn consistent features. The table below shows the linear probing accuracies of DINOv2 ViT-S on the noisy ImageNet-100 test set, when trained for 200 epochs on 100% noisy data (from Table 1(a)) versus on 50% noisy + 50% denoised data. We observe a significant drop of 7.4 points in accuracy when noisy and denoised data are randomly mixed during training.

In Appendix B.4, we comprehensively tested the performance of DINOv2 when trained and evaluated on mixed noisy + clean dataset, as seen in Table 7, introducing a small amount of clean images (e.g., 2%, 10%) during training destabilizes the process and slightly degrades performance. The above results shows naive mixing is suboptimal, and reinforces the necessity of using a staged curriculum learning approach that separates noisy and clean (denoised) images.  
 

Question 3: Can the method fine-tune a well-pretrained official DINOv2 model?

We sincerely appreciate your understanding of the computational constraints. DINOv2-small in the original DINOv2 paper is based on the closed-source LVD-142M dataset, which contains 142 million images. In contrast, our model is trained on ImageNet-1k with only 1 million images, which helps explain the performance gap. Additionally, the official DINOv2 models use a patch size of 14, resulting in more parameters and higher input resolution compared to our model, which uses a patch size of 16.

Yes, the proposed method, in particular, NCT can effectively fine-tune a well-pretrained DINOv2 model against noisy data. While our method is primarily designed for pretraining from scratch, we demonstrate that our denoised-regularized loss can significantly improve upon the vanilla DINOv2 fine-tuning baseline. We use the official ViT-B/14 weights, and fine-tune them on ImageNet-1k with Gaussian-255 noise for 4 epochs due to the limited rebuttal time window. Two finetuning strategies are employed: 1. Finetune directly using the original DINOv2 loss, 2. Finetune using our NCT loss where a copy of the pretrained weights serves as the denoised teacher. The linear probing accuracies on both noisy and clean test sets are shown in the table below:

We observe that NCT loss achieves significantly higher accuracy in both noisy target set (+8.0) and clean set (+10.2) compared to original loss. This shows the regularization not only helps convergence on the noisy set, but also reduces distributional shift from the clean set. We acknowledge that tuning on very noisy data will inevitably reduce clean performance due to the significant distributional gap, but researching how to improve noise robustness of pretrained models without sacrificing the clean accuracy remains a promising direction for future work.

We would like to emphasize that due to rebuttal time constraints, the above results were produced out-of-box without any cross-validation or hyperparameter tuning. We believe that extending the number of training epochs would further improve performance. We would also like to highlight that fine-tuning the official DINOv2 weights is non-trivial, as they only release weights for the backbone, but the iBOT and DINO heads are not released, which are crucial for fine-tuning. Thus, we first train the heads on clean data for 75k steps with the backbone frozen, before fine-tuning on noisy data. These results likely represent a lower bound, and we believe that access to the official iBOT and DINO heads, which are trained on large-scale datasets, would substantially improve fine-tuning performance.  
 

Question 4: Evaluation on real-world noisy datasets

Thanks for the valuable suggestion. The source “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations” (i.e., ImageNet-C) also utilizes synthetic noise injection processes. In our work, we actually employ the code base of ImageNet-C to create noisy datasets.

As acknowledged in Lines 354–356 of Section 5 (Limitations), a key challenge is the scarcity of large-scale real-world noisy datasets with reliable labels, which hinders standardized evaluation. As a promising next step, we are actively exploring the curation of such a benchmark to facilitate more realistic and comprehensive assessment. We fully agree that such a benchmark allows a fairer and more practical assessment of our method.  
 

Question 5: PCA Visualizations

Thanks for the great suggestion. We agree that PCA visualizations provide an intuitive and vivid representation of a model’s learned features. However, due to the current NeurIPS rebuttal policy, which prohibits any links, we cannot post images here. Instead, we will describe the PCA visualizations verbally.

We strictly follow the style in the original DINOv2 paper to generate PCA visualizations. Under Gaussian-255 noise, we observe that the DINOv2 baseline generates very noisy visualizations, as it struggles to separate salient objects (i.e., birds) from the background. In contrast, NC and NCT generate visualizations that clearly separate the salient objects with sharp boundaries. Notably, objects of the same class are consistently represented using similar color patterns across different images, indicating semantically aligned features. This helps explain why NC and NCT substantially outperform the DINOv2 baseline in downstream tasks. Moreover, from the visualizations, we can infer that NC and NCT will outperform DINOv2 not only in classification tasks, but also in dense tasks like segmentation and depth estimation. We will ensure to include these visualizations in the Appendix of our revised paper.  
 

Question 6: Releasing code

We have included all source code in the zipped supplementary materials, along with detailed instructions and links to download trained weights, which we will ensure are released to the community. We fully agree that open-sourcing the code and weights are beneficial to the scientific advancement of the community.  
 

We truly appreciate the feedback and believe the changes have made our submission much stronger. We hope we've covered all your questions, and we will remain available for conversation until the discussion period ends.

We sincerely thank Reviewer BtzQ for the thoughtful and highly positive review. Below, we would like to address each of your questions in detail:

Question 1: How does the performance of DINOv2 w/ NC degrade if a less effective denoiser is used?

Thanks for raising this practical question. To investigate it, we conducted additional studies on how the number of denoiser training epochs impacts the final SSL model’s performance. We use the number of training epochs as a quantifiable proxy for denoiser effectiveness. The evaluation follows our default setting: 200 epochs of DINOv2 ViT-S training with Gaussian-100 noisy data. The results are shown in the table below:

Performance of DINOv2 w/ NC on ImageNet-100 with the N2N denoiser trained for varying epochs

We observe that even training the N2N denoiser for just 1 epoch leads to a substantial improvement of DINOv2 w/ NC over the DINOv2 baseline (i.e., 10.7 increase), while being only 2 percent less than that of a well-trained denoiser (100 epochs). Although the 1-epoch denoiser still produces many undesirable artifacts like a very weak denoiser, the curriculum still yields substantial improvements in downstream performance. This highlights the robustness of our curriculum, which has considerable tolerance to the denoiser’s quality. A denoiser has to be unreasonably bad (e.g., training diverged or collapsed) to provide no return in performance. We will include this result as an additional ablation study in the Appendix B of our revised paper. Thank you again for the insightful suggestion; we believe this investigation further strengthens our contribution.  
 

Question 2: Any method to automate the selection of the restart epoch?

Thank you for raising this important point. We agree that the restart epoch is a key hyperparameter. In our experiments, we fix the total training budget (e.g., 200 epochs), which makes automated selection of the restart point non-trivial: restarting too early leaves insufficient time to benefit from clean supervision, while restarting too late limits the model’s capacity to adapt to noisy data. The noise level, model size and learning rate schedule can also impact the convergence speed in both stages. In practice, we heuristically select the restart point by beginning from the midpoint and adjusting based on performance trends observed in preliminary runs.

That said, if the training budget were unconstrained, we believe a dynamic transition based on model convergence, such as monitoring linear probe accuracy on the denoised data, would be a highly promising heuristic. We view this as a valuable direction for future work and will mention it as a potential improvement in Section 5.  
 

Question 3: Any plans to evaluate on real-world datasets? And how do you anticipate its performance will translate from synthetic to real noise?

We thank the reviewer for this important question. Indeed, as we acknowledged in Section 5, our current benchmark is limited to synthetic noise due to the scarcity of large-scale, labeled noisy datasets that support standard downstream evaluation. However, we fully agree that demonstrating robustness on real-world noise is critical for practical impact.

Yes, we do have plans to evaluate on real-world datasets. As a promising next step, we are actively exploring the curation of a large benchmark that contains real-world noisy images with accurate labels. Our goal is to include a diverse collection of noise types across domains such as biomedical imaging, astronomy, and remote sensing, where noise is often complex and structured. We believe this would provide a more rigorous testbed for evaluating our methods.

Furthermore, we are confident that the performance gains observed on synthetic noise will translate to real-world settings. Firstly, Our method has been validated on a wide spectrum of synthetic noise types (Gaussian, Shot, Speckle), noise levels (SNR from 9.2 dB to 0.3 dB), and multiple architectures (e.g., SimCLR, MoCo v3, DINOv2, iBOT). The consistent improvements across these settings indicate that the curriculum-based robustness is not tightly coupled to any particular noise assumption. Secondly, as shown in our response to reviewer 5arj comment 2, our method continues to perform well under mixed noise (i.e., Poisson-Gaussian), which is a common real image degradation. This supports the hypothesis that robustness induced by our denoised-to-noisy training strategy can transfer to more complex noise distributions.  
 

Question 4: Could the two-stage curriculum acts as a powerful regularizer, forcing the model to learn general features?

Thank you for the insightful observation. We fully agree that, in addition to mitigating information loss from explicit denoising, the two-stage denoised-to-noisy curriculum can indeed act as a powerful regularizer. By exposing the model sequentially to denoised (low-entropy) and noisy (high-entropy) data distributions, the training process encourages the emergence of more general and robust features, as the model must reconcile representations across varying input conditions. This interpretation aligns well with our motivation for the curriculum design and supports our findings in Section 4.4, where DINOv2 w/ NC and NCT generalize better than N2N + DINOv2 on clean validation sets. We will incorporate this valuable perspective into Section 4.4 of our revised version to provide a more complete discussion.  
 

Question 5: Why is the denoised teacher at 100 epochs a better regularizer than that at 30 epochs? And How is the embedding alignment preserved?

Thanks for pointing out this confusion. Here are our more detailed explanations. On lines 295-296, we explained that “the denoised teacher at this early stage lacks sufficiently strong representations to serve as an effective regularizer.”, this is because we empirically observed that the linear probing accuracy of the denoised teacher at 30 epochs is only 53.5 when tested on denoised dataset. This means that 53.5 would be an approximate upper bound of DINOv2 w/ NCT if using the 30-epoch denoised teacher, because DINOv2 w/ NCT consistently converges to or very slightly outperforms its anchor denoised teacher’s accuracy, as shown in table 1 (a) and line 285. To push for stronger performance and raise the ceiling, we thus employ denoised teacher at 100 epochs (i.e. accuracy = 57.2) to regularize the noisy training. And indeed, our experimental results show substantial improvement of NCT over both NC and DINOv2 baseline, and we expect its accuracy to reach 57.2 given long enough training epochs.

As for how the embedding alignment is preserved, on line 298, we explained that “the frozen teacher and the DINOv2 backbone are derived from the same training run and pass through the same 30-epoch state”. This means at 30 epoch, the frozen teacher and the trainable teacher share identical weights. Recent SSL literature [1][2][3] shows that self-supervised models evolve their representations in a coarse-to-fine manner: broad semantic clusters emerge early, while later training improves fine-grained separability. Consequently, although we continue to train the denoised teacher to full 100 epochs, its embeddings are generally still aligned with the backbone at 30 epoch state, as the coarse semantic structure (e.g., principal axes in embedding space) is preserved despite the continual training. Our empirical result also agrees with the theory as NCT solidly surpasses NC, demonstrating the 100-epoch denoised teacher can act as an effective regularizer during noisy training. We will make sure to add more discussion in section 4.3 in our revised manuscript.

[1] Reverse Engineering Self-Supervised Learning, Ben-Shaul et al., NeurIPS 2023

[2] On the Stepwise Nature of Self-Supervised Learning, Simon et al., ICML 2023

[3] Understanding Learning Dynamics of Neural Representations via Feature Visualization at Scale, Kuntala et al., NeurIPS Workshop 2023  
 

Comments on Computational Cost

Thank you for your suggestion. We agree that our pipeline introduces a slight increase in upfront computational cost due to training a denoiser and preprocessing the dataset. However, this additional cost is minimal compared to the full pretraining and downstream application stages. The denoising step only requires a single round of inference, and training the denoiser is highly efficient, as demonstrated by our ablation studies in Question 1. Moreover, in industry settings, inference cost increasingly becomes the primary bottleneck. A core contribution of our work is that it does not increase inference cost, which makes it particularly well-aligned with real-world deployment needs. Overall, we believe that the small upfront cost is a worthwhile trade-off for the substantial gains in inference efficiency and robustness. We will include this important discussion in Section 5 of our revised paper. Thank you again for pointing it out.  
 

We sincerely thank you for your feedback. Your suggestions have significantly helped us strengthen our submission, and we will remain available during the remaining discussion period.