FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed

We sincerely thank Reviewer keDD for the insightful questions. We would like to give detailed responses to each of your comments and questions.

Question 1. The experimental results can hardly support their claims:The final performance of the 112-224 curriculum (78.44%) is lower than the original DINOv2 (78.6%) for linear probing on IN100. A lower performance on dense tasks in Table 5.

We thank the reviewer for this important observation. We understand the reviewer’s concern regarding the final performance of the 112–224 curriculum. While performance improvement is valuable, the central contribution of our work lies in maintaining accuracy and robustness while significantly reducing training cost, and in providing a comprehensive analysis of frequency bias in the resulting model.
With only a 0.2% drop in accuracy, the 112–224 curriculum achieves both a robustness gain and a substantial reduction in training time. As noted by the reviewer, the dense task results in Table 5 further support this: FastDINOv2 achieves a 0.21% improvement in mAcc with only a 0.04% drop in mIoU, indicating comparable performance across both image-level and pixel-level tasks.
To further assess the effectiveness of our method on downstream applications, we also conducted additional experiments on instance-level recognition using the Oxford and Paris datasets, each evaluated with three difficulty-level splits. On the Paris dataset, FastDINOv2 achieves comparable performance across all levels. On the Oxford dataset, our model shows substantial improvements on the easy and medium splits, further demonstrating the robustness and generalization of our approach.

Question 2. The conclusion is made on ImageNet-100, an easy subset with less noise. For large-scale datasets full of diverse noise, the impact of Gaussian noise patching is questionable.

We thank the reviewer for raising this important point regarding the potential impact of training on larger and noisier datasets. In Table 1, we provide results for both linear probing and robustness evaluation on ImageNet-1K and ImageNet-C. These results show that FastDINOv2 significantly reduces training time, with only a slight drop in linear probing accuracy and comparable robustness—demonstrating that the proposed data curriculum remains effective even at a larger scale. The maintained robustness further suggests that Gaussian Noise Patching continues to offer benefits on more diverse datasets.
Due to computational constraints, we were unable to conduct additional experiments on ImageNet-22K, but we plan to do so once we have access to sufficient resources.

Question 3. How to combine masked image modelling and Gaussian noise patching is not clear. Does the GP applied to visible patches only? Or a replacement of masking?

We thank the reviewer for bringing up this important aspect. To clarify, Gaussian noise is applied to a randomly selected region of the image with a random size to form a Gaussian noise patch. This process is independent of patch visibility in masked image modeling and is not intended as a replacement for masking. Instead, we introduce Gaussian Noise Patching as a complementary technique designed to help mitigate the frequency bias introduced by the data curriculum.

Question 4. A similar strategy is popular in progressive training, which increases the resolution and the augmentation strength gradually during training [1,2]. As mentioned in [1], low-resolution does not mean an east task in MAE. In contrast, low-resolution images could act as regularisation to improve performance[3].

We understand the reviewer’s question regarding the novelty of our work. One of our key contributions is a comprehensive analysis from the perspective of frequency and robustness, specifically focused on models trained with a low-frequency data curriculum. To the best of our knowledge, we are the first to identify and characterize the frequency bias introduced by such a training framework. Furthermore, we demonstrate that the combination of low-frequency training and Gaussian noise patching forms a non-trivial curriculum, which leads to more balanced and robust performance compared to using either technique in isolation. In addition to maintaining or improving downstream task performance and mitigating spurious correlations, our method achieves significant reductions in training time specifically for the DINOv2 model—an outcome not demonstrated in prior work.
The reviewer refers to [1] and point out that low-resolution images act as a form of regularization rather than representing easier samples. From our perspective, this interpretation is not necessarily in conflict with defining them as “easy.” The self-attention mechanism in ViT inherently functions as a low-pass filter—it emphasizes low-frequency components while attenuating high-frequency signals. From this viewpoint, downsampled images align more closely with the model's internal bias and can thus be considered easier for the model to learn from.

Question 5. Why use fewer training epochs? To show the speedup number, the final performance should match the original number.

We thank the reviewer for pointing out this confusion. In our evaluation, the speedup is measured by comparing training time while keeping the final performance comparable. FastDINOv2 converges more quickly with the data curriculum, and in our experiments, it reaches the same level of accuracy as the baseline using fewer epochs. This faster convergence demonstrates the efficiency of our approach in reducing training cost while maintaining final performance.

We greatly appreciate your thoughtful follow-up. Upon reflection, we recognize the importance of figures illustrating convergence trends and comparing our method with the baseline. However, the current NeurIPS rebuttal policy prohibits posting links or images. Instead, we provide the following table comparing the training time required by DINOv2 and FastDINOv2 to reach specific linear probing accuracy levels. Training time is measured in NVIDIA L40S GPU hours:

During the low-frequency training stage (when FastDINOv2 reaches 55% and 65% accuracy), FastDINOv2 achieves similar convergence speed to the baseline trained on full-resolution images. This is attributed to the inherent low-frequency bias in deep neural networks and transformers, such that the model is more inclined to learn low-frequency information first during training. Furthermore, this low-frequency stage provides a strong foundation for the subsequent full-resolution training stage (when FastDINOv2 reaches 75% and 78% accuracy), enabling FastDINOv2 to more efficiently and effectively learn fine-grained, high-resolution details from full-resolution images.

We sincerely thank the reviewer for recommending this analysis—we believe this additional demonstration strengthens the contribution of our work. We will remain actively engaged in the remaining discussion period for additional comments.

We sincerely thank Reviewer Yv4e for the valuable comments and positive review. We would like to address each of your questions in detail:

---

Question 1. Could you provide any intuition or results from preliminary experiments that led to the choice of 75/25? How does the model perform with, for instance, a 50%/50%, 90%/10%, 0%/100% or 100%/0% splits? What led to your decision of 75/25?

We thank the reviewer for this essential question. For the ablation experiments on curriculum ratios, we used a fixed number of training epochs to ensure a fair comparison across different splits. Within this controlled training budget, our goal was to improve both learning efficiency and final performance through a two-stage curriculum.
This design addresses two key aspects of our method: (a) leveraging the efficiency of low-resolution training, and (b) preserving final performance through progressive exposure to high-frequency content. The first stage uses downsampled (low-resolution) inputs that emphasize low-frequency information, while the second stage with high-resolution images enables learning of fine-grained details. If the first stage is too short, the model fails to fully benefit from this simplified setting. If it is too long, the model lacks sufficient exposure to the full-resolution features necessary for downstream performance.
We experimented with multiple splits on ImageNet-100 with ViT-S, including 65/35, 75/25, and 85/15. Among these, the 75/25 split consistently yielded the best trade-off, achieving the highest linear probing accuracy. More extreme cases such as 0%/100% would have no acceleration effect, and 100%/0% would have only low-frequency data training, which performed notably worse than the above splits.

This 75/25 curriculum design also enables cost-effective pretraining. By using downsampled images during the initial stage, GPU memory usage is significantly reduced, making it feasible to perform the majority of pretraining on smaller GPUs.

---

Question 2. In table 2 there is a fairly consistent trend, with one outlier. Do you have any insight into why model performance degrades as the image patches get smaller in all cases except the 128-224 which is worse than 112-224?

We thank the reviewer for this careful observation. Indeed, Table 2 shows that as the patch size of the low-resolution images decreases, model performance tends to degrade. This is expected, as smaller image patches discard more high-frequency information, limiting the model's ability to learn fine-grained features during the early phase of training.
The 128-224 case performs slightly worse than 112-224, and we believe this minor deviation is due to a tradeoff between resolution and curriculum duration. Specifically, the 128-224 setting may not sufficiently reduce the input complexity to yield a significant efficiency benefit. In contrast, the 112-224 split achieves a better balance—providing just enough simplification to aid convergence while preserving meaningful structure.

---

Question 3. Do you think framing your model as competitive with the standard dinov2 performance is accurate? Or is your paper better framed as a slight trade off in accuracy but improvements in speed?

We thank the reviewer for this important clarification. Upon reflection, we agree that describing our model as “competitive” with the standard DINOv2 performance is somewhat overstated. One of our primary focuses lies in improving training speed and reducing computational resource requirements. Thus, it is more accurate to describe our model as a slight tradeoff rather than strictly competitive. We appreciate the reviewer pointing this out and will clarify this distinction more explicitly.

We sincerely thank Reviewer PgPD for the positive review and important suggestions. Below, we would like to address each of your questions:

---

Question 1. The fixed 75% / 25% split for the curriculum is a key design choice. Have you investigated other splits or considered adaptive strategies where the transition point is determined dynamically based on convergence metrics?

We thank the reviewer for this important question and for recognizing the limitations involved in this design choice. We have indeed experimented with different splits, including 65/35, 75/25, and 85/15, and found that the 75/25 split yields the best performance.
From a training convergence perspective, it is essential that the model can benefit sufficiently from both stages in our curriculum—low-frequency and full-resolution training. If the first stage is too short, the model may not develop a sufficiently strong initialization and gain the convergence speed up; if the second stage is too short, it may not fully learn more fine-grained details from training on high-resolution data.
Given that the total number of training epochs is fixed in our current setting, we adopt a heuristic 75%/25% split for FastDINOv2. However, we agree that dynamically determining the transition point based on convergence metrics, such as online linear probing accuracy, could further optimize performance. We see strong potential in extending our method to include such adaptive curriculum strategies.

---

Question 2. In the introduction (line 89) and elsewhere, you describe the curriculum as inducing a "high-frequency bias". This is not very clear. Would it be more precise to state that the model develops higher robustness to low-frequency corruptions, which implies a more substantial reliance on high-frequency features? Could you further elaborate on this causality?

We thank the reviewer for bringing up this thoughtful and important question. It is indeed more precise to describe the observed effect as increased robustness to low-frequency corruptions and a greater reliance on high-frequency features. This phenomenon arises from the intrinsic frequency biases of deep neural networks (DNNs), Vision Transformers (ViTs), and property of curriculum learning.
Prior work has shown that DNNs exhibit a spectral bias—they tend to learn low-frequency components earlier in training and gradually incorporate higher-frequency features as training progresses [1,2]. Additionally, the self-attention mechanism in ViTs behaves like a low-pass filter [3], further reinforcing this early preference for low-frequency information.
In our proposed curriculum, we exploit these inherent biases by explicitly organizing training to begin with low-frequency components, which serve as easier examples, and then transition to full-resolution images, which contain more high-frequency content and are thus harder. This progression aligns with findings in curriculum learning, where beginning with easier examples can smooth the optimization landscape and enable more effective learning of complex patterns later on [4].
As a result, once the model has built a strong foundation from low-frequency features, it is better positioned to learn and rely on high-frequency components during the later stages of training. This leads to improved robustness against low-frequency corruptions, as the model has developed stronger sensitivity to the more informative high-frequency features.

---

References

[1] Rahaman, N., Baratin, A., Arpit, D., et al. (2019). On the Spectral Bias of Neural Networks. Proceedings of the 36th International Conference on Machine Learning (ICML).
[2] Xu, Z.-Q. J., Zhang, Y., & Xiao, Y. (2019). Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523
[3] Wang, Peihao, et al. Anti‑Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice. ICLR 2022, arXiv:2203.05962, 2022.
[4] Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning (pp. 41–48). ACM.

---

Question 3. Bicubic interpolation was used for downsampling the images. Did you study the impact of other low-pass filtering methods (e.g., a Gaussian blur before downsampling) on the efficiency and robustness of the final model?

We thank the reviewer for this insightful question. Our initial hypothesis was that reducing high-frequency content in images could improve both convergence speed and model robustness, as low-frequency information typically captures the most generalizable features hence “easier”. To this end, we applied bicubic downsampling, which inherently acts as a low-pass filter and emphasizes low-frequency components.
We also considered whether further reducing high-frequency signals with additional filtering such as Gaussian blur might improve performance. However, our experiments showed that bicubic downsampling alone was sufficient to retain the relevant low-frequency information while maintaining a balance with spatial detail. Additional filtering methods, including Gaussian blur and anti-aliasing, resulted in slight loss of detail and degraded model performance.
Therefore, we concluded that downsampling with bicubic interpolation provides an effective trade-off between low-frequency extraction and information retention, and we adopted this approach in our final setup.

---

Question 4. Given the lack of error bars, would it be feasible to run at least a few training runs with different random seeds for a smaller configuration (e.g., ViT-S on ImageNet-100) to provide at least a sense of the variance and stability of the results?

We thank the reviewer for understanding our limitation on computation resources and time. We conduct experiments using multiple random seeds (specifically, seeds 0, 10, and 5) on a smaller configuration—ViT-S trained on ImageNet-100 for FastDINOv2 with the 112-224 curriculum—and tested on ImageNet-100 validation set and ImageNet-100-C. While the output naturally varies across seeds, the overall performance remains consistent. These results support the robustness of our main findings.

We sincerely thank the reviewer PiGo for their questions and comments. Below, we answer each question accordingly:

---

1. Experiments are performed solely on DINOv2 (ViT). Will the curriculum show similar results for other methods? Is it also viable for CNNs?

We thank the reviewer for bringing up the important issue of generalization across frameworks and model architectures. To address this, we conducted additional experiments by training a SimCLR model with a ResNet backbone using our FastDINOv2 method (denoted as “FastSimCLR”). Using the LARS optimizer, the SimCLR baseline was trained for 200 epochs on full-resolution images from ImageNet-100, while FastSimCLR was trained for 150 epochs on low-frequency data followed by 50 epochs on full-resolution images. Despite the differences in inductive biases between ViTs and CNNs, FastSimCLR shows faster convergence. Furthermore, combining this curriculum with Gaussian noise patching improves robustness. We thank the reviewer again for this insightful suggestion–we believe this experiment further demonstrates the effectiveness of our method.

---

2. The experiments are performed mostly on an ImageNet subset (ImageNet-100). Concerns about how the curriculum performs on larger datasets (ImageNet-1K, ImageNet-22K, etc.)

We thank the reviewer for this insightful question. We agree that demonstrating the effectiveness of our method on larger datasets is crucial. Due to limitations in computational resources, we were unable to conduct experiments on ImageNet-22K. However, we did evaluate our approach on ImageNet-1K, as presented in Table 1. In this setting, we trained a ViT-B model on ImageNet-1K, evaluated its representations via linear probing on the ImageNet-1K validation set, and tested robustness on ImageNet-C. Our results show that FastDINOv2 achieves a substantial reduction in computational cost while maintaining performance comparable to the baseline DINOv2. Notably, the improvements in convergence speed and robustness observed on ImageNet-100 carry over to ImageNet-1K, suggesting that our curriculum design generalizes well as dataset scale increases. These findings imply that our method may also scale effectively to even larger datasets such as ImageNet-22K. With additional computational resources, we intend to validate our approach at the ImageNet-22K scale.

---

3. The paper does not compare to competing frameworks.

We appreciate the reviewer’s concern regarding comparisons to other efficient training frameworks. Many existing approaches are tailored to specific models—such as [1] or [2]—and cannot be applied directly to DINOv2 without significant changes to training configurations. Moreover, more general frameworks like [3] do not observe a speed-up for DINO in their experiments. To our knowledge, our work is the first to propose an efficient training pipeline for DINOv2 based on a data curriculum strategy.

---

References

1. Li, Runze, et al. RECLIP: Resource-Efficient CLIP by Training with Small Images. arXiv preprint arXiv:2304.06028, 2023.

2. Wu, Jiantao, et al. DailyMAE: Towards Pretraining Masked Autoencoders in One Day. arXiv preprint arXiv:2404.00509, 2024.

3. Wang, Yulin, et al. EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Training. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, 2024, pp. 8036–8055.