You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning

We thank the reviewer for their positive reception of our work. In the following we will be addressing each of your comments.

W1) Please refer to the general rebuttal. We add an analysis on MOCOv3 (a more modern version of the SimCLR paper) and explain why our choice is constrained to DINOv2 (the Crop setup was unstable with MOCOv3 and we didn't manage to train a model with it). Our claim might hold for other methods as long as it’s possible to scale them effectively to ImageNet-22k (which is not an easy task). In the meantime, we think that our paper can already impact people training DINOv2 models on other modalities [1,2,3,4,5,6].

Q1) It’s an interesting question about insights on reconstruction-based SSL. Our paper shows that the constraint on hand-crafted data-augmentations is not the main factor in the existing gap, but multiple factors can still explain it. First, our model uses DINOv2, which is carefully optimized to work on large scale datasets. I-JEPA, however fails to train good models on ImageNet-22k, but according to the results in our paper, this could be explained by the use of suboptimal hyper-parameters. MAE also fails to scale effectively, in line with a new paper explaining that learning by reconstruction forces the model to learn uninformative features [7]. At the end, the issue might be that we add too much unused information to learn for reconstruction-based methods, and that scaling data increases even more this amount of information.

[1] R. J. Chen et al. Towards a general-purpose foundation model for computational pathology [2] H. Xu et al. A whole-slide foundation model for digital pathology from real-world data [3] E. Vorontsov et al. Virchow: A Million-Slide Digital Pathology Foundation Model [4] T. Moutakanni et al. Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning [5] F. Perez-Garcia et al. RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision [6] J. Tolan et al. Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar [7] R. Balestriero et Y. LeCun, Learning by Reconstruction Produces Uninformative Features For Perception

We would like to thank you for recognizing the importance of scaling SSL on other domains. In the following we extended our experiments, including also other imaging domains, making our conclusions more solid.

W1) You indicated that the main motivation for this work, which is to allow using SSL models on different domains, was not supported by experimentation. Indeed, generalization to new domains is very important given that multiple papers used DINOv2 on medical imaging, remote sensing, and many other ones, either directly as is or as a pretrained model to finetune [1,2,3,4]. We decided to test (Table 1 in the rebuttal) our models on such domains where invariance toward domain-specific augmentations could be harmful:

• logos, signs and product’s packaging where for example text can be degraded by blurring and zooming, and arrows can be inverted if the features are invariant to the flipping;
• remote sensing, where colors and zooming can have a big impact on the performances;
• medical imaging where classes differ in type (shape vs texture) and size (small nodule, broken ribs, enlarged heart etc.).

Table 1 in the rebuttal shows that a model trained without domain-specific augmentations tends to generalize better on most of these domains, notably on products (RP2K and Products-10k), remote sensing (FireRisk) and medical dataset (MIMIC, CheXpert, Vindr and NIH-14).

We gained 4.8% acc. on GTSRB when switching from the original DINOv2 to the Crop only one, showing the impact of resizing on this task.

On RP2K, Products-10k, and medical imaging datasets, all methods without domain-specific data-augmentations get, in general, better performances, proving that those augmentations were tailored for ImageNet-like tasks but can be detrimental to other ones, as stated in [8].

W2) Thank you for the comment. Indeed, models might not learn invariance even when trained with data-augmentations. However, we quantitatively show in Table 3 in the rebuttal that the Original setup is more invariant than the Shared, Crop+Resize and Crop setup toward data-augmentations. For that, we follow the prior work of [5, 6] and use as a metric the average cosine similarity between augmented views of the same image. We also normalize based on the average similarity between random images to take into account potential model variability.

Analyzing the quantitative results of our empirical study in Table 3, we observe an interesting pattern, with the invariance decreasing between the Original, Shared, Crop+Resize and Crop settings. It was expected that the most invariant setup would be “Original”, and the least invariant would be “Crop”, but it’s interesting to see that the Shared and Crop+Resize setup have different levels of invariance despite having the same “invariance enforcement” in the loss. A more detailed analysis of data augmentations and equivariance would be a good follow-up to this paper.

W3) The gap between low and large data regimes cannot be recovered only by training for longer on the smaller dataset. As it is discussed in this study, scaling SSL is harder than only changing the dataset’s size. Increasing the number of epochs alone will lead to overfitting. This happens less with data-augmentations as they artificially increase the dataset size. Increasing the number of data alone will help, but the training will be suboptimal as the model will see each image for a shorter amount of time. Thus, scaling both training data and length is what can make models with and without data augmentations reach the same performances.

Another remark about the remaining gap for our 500 epochs setup with LVD-142M with and without augmentations. We shortly mentioned this in the limitations, but here is a more detailed explanation:

• Hyperparameters were tuned by DINOv2’s author using all data-augmentations. We used the exact same parameters for all our setups, and thus are probably using suboptimal parameters when removing augmentations.
• The larger the model used, the more data we need to close the gap. You can see this phenomenon in the paper, Figure 3 on the right. We might be in a setup (ViT-L) where more data is needed to fill the gap (for example, CLIP ViT-L models are trained using billions of images [7]).
• We don’t use data-augmentations in our linear classifier training to remove confounding factors when evaluating our models. Adding data-augmentations here isn’t necessary to reach best performances using the original DINOv2. However, some invariances are actually good on ImageNet-like domains [8], and the ones used in the original setup are tailored exactly for this. The gap might close even more if we optimized data-augmentations during the classifier training, transferring the invariances from the pretraining step to the classification step.

Questions: Sorry for the confusion about AIM and DFN-2B+. These abbreviations come from the paper [9]. We will add all citations to Table 1.

[1] X. Song, General Purpose Image Encoder DINOv2 for Medical Image Registration

[2] M. Baharoon et al. Towards General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks

[3] B. Cui et al. Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery

[4] X. Bou et al. Exploring Robust Features for Few-Shot Object Detection in Satellite Imagery

[5] Q. Garrido et al. Learning and Leveraging World Models in Visual Representation Learning

[6] A. Devillers & M. Lefort, EQUIMOD: an equivariance module to improve visual instance discrimination

[7] M. Cherti, Reproducible scaling laws for contrastive language-image learning

[8] I. Bendidi et al. No Free Lunch in Self Supervised Representation Learning

[9] A. El-Nouby et al. Scalable Pre-training of Large Autoregressive Image Models

We would like to thank the reviewer for recognizing the importance of the role of scale in self supervised learning and for their questions that made us think more deeply about the semantics and experiments we used throughout our paper. In the following we will be addressing each of your comments.

W1) Please refer to the general rebuttal. Thanks to your comment, the title of our paper will be changed to: “Self-supervised learning doesn’t need domain-specific data augmentations at scale” to highlight more the claims of the paper.

W2-3) Hand-crafted data-augmentation is the term used in the original I-JEPA paper [1] to talk about most augmentations but masking and RandomResizedCrop. Hand-crafted data augmentations are mentioned in I-JEPA as augmentations that can’t be used on other modalities (thus hand-crafted for the specific modality in use). In our paper, we used the same term but included the Resizing as a hand-crafted augmentation as it isn’t usable in text or modalities where resampling is impossible or doesn’t make sense.

We propose to change “hand-crafted” to “domain-specific” to make it more explicit. We think that this change in semantics (we used hand-crafted data augmentation in the paper but not the original title) answers both of your points about the boldness of our claim (that was, we think, only in the title) and your question about the fact that cropping and masking are augmentations. We will also make sure to update the paper with the correct wording when necessary.

To support our claims, we actually don't need our model to be trained with exactly zero invariance or data augmentations. A lot of papers are claiming that data augmentations are necessary to achieve good performance with classical SSL [1,2,3] and that it’s not the case for reconstruction-based methods like MAE, I-JEPA or AIM [4,5]. However, MAE, I-JEPA, and AIM still use data augmentations (masking and RandomResizedCrop). It’s also worth noting that all the data-augmentation ablations in [1] are only done on what we call “domain-specific augmentations” and that all their setup include RandomResizedCrop. We show that their results don't hold at scale with DINOv2, which we think is novel and interesting.

We claim that we can remove all domain-specific augmentations and still train a very powerful model, using fewer augmentations than all the other methods in the literature. When we analyze the impact of data-augmentation and invariance (Table 3 in the rebuttal), we don’t need it to be exactly zero for our claims to hold. We just need the Shared and Crop+Resize setup to enforce less invariance than the Original setup. The performance gap that is created by enforcing less invariance is compensated by increasing the amount of data, showing that the main bottleneck was the artificial increase of data, not the invariance making. The impact of the RandomCrop and masking augmentations doesn’t change between all our setups, and we only analyze the difference relative to all the other augmentations.

We also want to add that we have a table in the original paper (Table 2) that ablates the masking augmentation, training a strong model with only one augmentation (RandomCrop, without resizing and without masking). This model still beats all the other baselines from Table 1 in our paper by a large margin.

W4) Please refer to the general rebuttal. We add an analysis on MOCOv3 (a more modern version of the SimCLR paper) and explain why our choice is constrained to DINOv2 (the Crop setup was unstable with MOCOv3 and we didn't manage to train a model with it). Our claim might hold for other methods as long as it’s possible to scale them effectively to ImageNet-22k (which is not an easy task). In the meantime, our paper can already impact people training DINOv2 models on other modalities [6,7,8,9,10,11].

[1] Morningstar et al, Augmentations vs Algorithms: What Works in Self-Supervised Learning

[2] T. Chen et al. A Simple Framework for Contrastive Learning of Visual Representations

[3] J.-B. Grill et al. Bootstrap your own latent: A new approach to self-supervised Learning

[4] M. Assran et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

[5] A. El-Nouby et al. Scalable Pre-training of Large Autoregressive Image Models

[6] R. J. Chen et al. Towards a general-purpose foundation model for computational pathology

[7] H. Xu et al. A whole-slide foundation model for digital pathology from real-world data

[8] E. Vorontsov et al. Virchow: A Million-Slide Digital Pathology Foundation Model

[9] T. Moutakanni et al. Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning

[10] F. Perez-Garcia et al. RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision

[11] J. Tolan et al. Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar