Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks

1. "How to introduce synthetic noise in ImageNet and YFCC15M needs more explanation. "

We mentioned some details of introducing synthetic noise in Appendix A.1. More specifically, for ImageNet, we first initialize a noise matrix with the pre-defined noise ratio, where each element denotes the probability of the ground truth label being corrupted to a noisy label. We then use the noise matrix to flip the ground truth labels for each class. For YFCC15M, since it is an image-text pair dataset, we randomly swap the text between two randomly sampled pairs to make the image and text unmatched. We conduct this pair swapping until the ratio unmatched pairs reach the pre-defined noise ratio.

2. "The pattern SVE analysis of the ImageNet model and YFCC15M model are slightly different in Fig.3, and perhaps need more explanation."

Thanks for this good question. The reason why SVE patterns are different for ImageNet trained model and YFCC15M trained model is mainly because of the different pre-training objectives, i.e. supervised cross-entropy and contrastive. This can be observed from the SVD spectrum plot of ImageNet models in Fig.10 and YFCC15M models in Fig. 12. The ImageNet R50 models, in general, have smaller SVD values and faltter SVD spectrums, compared to YFCC15M models on downstream datasets. This why the SVE of ImageNet models in Fig.3 are narrower in x-axis range and wider in y-axis range, compared to YFCC15M models. For better interpretation, we have included a zoom-in version of Fig.3 in Appendix.

3. "Since ImageNet or YFCC15M itself also originally contains noise, is there any optimal noise ratio that achieves the best ID downstream performance? "

We totally understand your point that there might exist some optimal noise ratio in pre-training data that achieves the best ID downstream performance. However, this is contrary to the goal of our work, we demonstrated that although slight noise can boost the ID performance, it is always detrimental to OOD performance, which inspires us to mitigate the effect of pre-training noise. Besides, finding the optimal noise ratio for ID tasks might be infeasible because it would require tuning and validation on different ID tasks, thus presenting different optimal values according to the task.

4. "Since NML assumes an inaccessible pre-trained model, how is other black-box tuning methods perform on the noisy model learning setting?"

We find most of the parameter-efficient tuning methods are not black-box. There is a recent black-box fine-tuning method termed BlackVIP, which has also been mentioned in our related work, and we provide some of its results here on Laion-2B CLIP pre-trained ViT-L compared to our method:

[1] Changdae Oh et al. BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning.

1. "This paper is featured with extensive empirical results. However, the core techniques in methodology (analysis and regularization of singular value spectrum ) have been studied in existing works[e.g. Chen et al., 2019, Bardes et al. 2022], which may undermine the theoretical contribution of this work."

• First of all, as we discussed in the paper, the proposed method is indeed inspired by some related works [e.g. Chen et al., 2019, Bardes et al. 2022]. However, our method are not a direct application of them but designs them as specific regularizations, which are more straightforward and simpler to mitigate the noise in pre-training from our analysis.
• Second, the noisy model tuning approach is only one aspect of our contributions. Our main contribution lies more in being the first pioneering work to identify and analyze the effect of pre-training noisy datasets on downstream tasks. We think it is more important to bring up the question and analysis to the community, and more future work could be done.

2. "Some figures are hard to understand by themselves. E.g. different types of marks are cluttered in Fig3."

Sorry for the confusion. We provide a zoom-in version of Fig.3 in Fig.10 of Appendix. We hope the zoomed region can have a better visualization there. Please refer to the general response.

3. "On what scale the SVE and LSVR is computed? The entire dataset?"

The SVE and LSVR are computed on the entire test set of each downstream task.

4. "Do the conclusions (Fig1- 3) always hold for stronger backbone models other than resnet50?"

Thanks for this good question. We believe the conclusions from Fig 1-3 will scale to larger backbone models because it is related to mainly the noise in pre-training data rather than the model capacity. Note that we follow the normal pre-training recipe with heavy regularization techniques such as stochastic depth, mixup, cutmix, label smoothing, etc. This also demonstrates that noisy data is the root of the conclusions, which has little relation to regularization. Similar observations also hold in [1].

However, due to the expensive computing resources that will be needed to pre-train larger and stronger backbone models, we won't be able to formally verify all of this during the rebuttal period. Especially for CLIP pre-training of Vision Transformers (ViT), it takes 3-4 days to pre-train on YFCC15M with 8 A100 GPUs and we need to train 5 models with different noise ratios of YFCC15M. Moreover, it is found in OpenCLIP that ViT in general, needs much more data to achieve better performance than R50, and training on YFCC15M yields worse performance than R50. We will include as many analysis results as we can for synthetic noisy ImageNet pre-training of ViT here during the rebuttal period, and provide the full results of both ImageNet and YFCC15M in future revision of the paper.

[1] Chiyuan Zhang et al. Understanding deep learning requires rethinking generalization.

5. "In the loss function, are the regularizations computed per batch?"

Yes, we computed the regularization per batch, with a batch size of 32. We found the proposed regularization terms not sensitive to batch size, as long as it is not too small (e.g. 4). We provide a brief ablation of batch size in our method on JFT-300M semi-supervised pre-trained EfficientNet-B3 here:

1. "I find Figure 3 challenging to interpret...reached by the authors based on this figure."

Sorry for the confusion in Fig. 3, and thanks for your suggestions on making Fig. 3 more clear. In fact, we have tried to connect the points in Fig. 3, but it only makes it more difficult to visualize and interpret. Instead, we have provided a zoom-in version of Fig. 3 (a) and Fig. 3 (c) in Fig.10 of Appendix. We hope this zoom-in version is more interpretable. In general, for the SVE of ID tasks, we first observe the increase of SVE as the noise goes up to 5%, with performance improvement for most of the tasks. Then we observe a further increase of SVE as noise increases to 10%, 20%, and 30%, and usually with performance drop.

2. "I've observed a potential contradiction between SVE and LSVR. For instance, in the case of two-dimensional features, [1.0, 0.0] exhibits the highest LSVR while having the lowest entropy. It would be beneficial if the authors could provide further clarification regarding this inconsistency."

Thanks for mentioning the view of two-dimensional features. However, there might be a little misunderstanding here, and we would like to provide more discussion.

• First, we need to clarify that both SVE and LSVR are computed from the singular values of the features on the downstream test datasets of size $M$. When $M=1$ with a single point, the non-zero singular value would be trivial as a scalar of the magnitude of its unit basis vector (this can be simply verified by NumPy). In this case, any single point would present highest LSVR with lowest entropy, but has no relation to our observation and anlysis.
• In general, we would require the size of the training samples and testing samples to be larger than 1 ($M > 1$), which is practical in real experiments.

3. "I would appreciate it if the authors could offer more detailed explanations as to why a slight amount of noise can benefit in-domain (ID) tasks. This is somewhat contradictory to the existing literature on learning with noisy labels, and additional insights would be valuable."

Thanks for this good question. We also find it interesting and more discussions should be added. We share our insights here:

• Why a little noise can help?

  • Slight noise in pre-training makes the model learn to fit those noise, where the extra dimensions in the feature space are utilized. Thus, when applied this model to a downstream task, its extra dimensions in feature space provide better initialized and discriminative features (from higher dimenstional space), leading to better downstream accuracy.
  • More noise in pre-training results in more dimensions to fit the noise. However, more dimensions to fit the noise is at the cost of less transferable features learned from the clean data and more nauce learned from the noise data, thus further increasing noise leads to worse performance.
  • We do not think the better performance on downstream of slight noise in pre-training is related to better regularization in our pre-training experiments. This is similar in [1], where even heavy regularization techniques are used, the model can still fit perfectly to random noise.

• The relation between noisy label learning and noisy model learning: This might look contradictory to learning with noisy labels, but our problem is indeed different from noisy label learning, so we give it a new name as noisy model learning. As shown in Fig. 4,

  • Noisy label learning: given a pre-trained model (or learn from scratch), the downstream datasets or any data we want to train the model contains noise.
  • Noisy model learning: given a pre-trained model trained on noisy datasets (which we cannot access, such as the black-box pre-training data of CLIP), we want to reduce such effect brought by the noisy pre-trained data on any downstream datasets.

That being said, our noisy model learning is naturally complementary to noisy label learning. We have an experiment in Sec. 4.3 to show that they can be used together.

Additionally, we can informally consider that noisy training data can lead to noisy weights, as in NoisyTune [2], where the authors also found introducing noise to pre-trained weights could help with downstream tasks, reaching similar conclusions as our study.

[1] Zhang et al. Understanding deep learning requires rethinking generalization. ICLR 2017.

[2] Wu et al. NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better. ACL 2022.

1. "Improvements are needed in the writing and presentation quality.", "Some claims in the paper are unclear and confusing."

We sincerely thank you for your careful proofreading of our paper. We have fixed the typos and made the confusing statement more clear in the revised paper.

2. "The authors did not mention the limitations of...why the methods might fail is crucial for practical applications"

Thanks for pointing this out. We briefly mentioned the limitation in the conclusion section (the last two sentences) due to the space limit. Now we discuss more limitation and potential failure here, which has been added in appendix C.1 and C.2. of the paper.

Limitation. The limitation mainly lies in our empirical study of the noise in pre-training. Due to the limited computing resources to extensively investigate different noise ratios by pre-training different models, we could only conduct our "motivational experiments" on ImageNet-1K and YFCC15M using ResNet-50 (which are already very expensive since we need to pre-train several ResNet-50 models on ImageNet or YFCC15M from scratch). Note that most of the SOTA foundation models are of much more parameters and are trained on much larger (and possibly inaccessible) datasets, which are beyond our reach to perform pre-training. Additionally, the empirical experiments are limited to actual supervised pre-training. While there are some existing studies showing the potential issue of noisy pre-training data, we could reveal more by scaling our experiments on larger backbones and datasets in the future.

Potential Failure. Yes, we agree that identifying failure cases indeed helps understand the problem better. We do observe some failure cases of the proposed methods. For example, from the results in Table 7, the proposed method falls short of LP on Caltech101 on almost all backbones we studied, while improving over MLP. Our hypothesis for the failure is that the SVD regularization term in the proposed method might need to optimize the top-$K$ singular values instead of just the largest one. The optimal value of $K$ might be different across datasets. However, setting $K=1$ can already achieve reasonable performance for most of the tasks, as shown in the results.

3. "Self-supervised pre-train does require external supervision...to self-supervised pre-trained models?"

Thanks for this wonderful question! We do believe self-supervised learning is worth further discussion:

• First of all, self-supervised learning does not require the explicit supervision of $Y$, but can be seen as supervised by the original input $X$, as we mentioned in footnote 2 of page 2 in our paper. In this context, it still requires supervision and the noise problem still exists: if $X$ is corrupted, which is usually the case in large-scale pre-training, it can be viewed as noise in self-supervision.
• Second, despite some previous research [1] indicating that the self-supervised pre-training data contains some noise, we give more examples here. It turns out that corruption in $X$ can also have different formats in the actual data. For instance, in images, the corrupted $X$ can be poisoned or attacked images that do not contain natural content, and it will become noise for masked image modeling pre-training. Taking contrastive learning, MOCO [2], as another example. MOCO is a instance discrimination task and the negative pairs could potentially contain images that are similar to the positive anchor. This can also be viewed as noise. In auto-regressive language models such as GPT series, the noise would simply wrong or repetitive words in the sentence.
• Third, our algorithm can naturally work for self-supervised pre-training models, as shown in Table 1 and 2, where most of these foundation models (ViT, CLIP, EfficientNet, GPT2, BERT, RoBERTa, and text-ada-002) are self-supervised pre-trained. Studying and understanding the corrupted $X$ (or noise) in self-supervised pre-training would also be very interesting and is our future direction. We believe the observation and conclusion from this work would scale and extend to the self-supervised learning scenarios because eventually, they fall into the same general formulation of noise in supervision.

In the future version of the paper, we will add more experiments and discussions about self-supervised learning.

[1] Yang et al. Xlnet: Generalized autoregressive pretraining for language understanding. NeurIPS 2019. [2] Kaiming He, et al. Momentum Contrast for Unsupervised Visual Representation Learning

4. "When you trained CLIP, did you train the text encoder together or you use a frozen text encoder?"

For CLIP pre-training, we strictly follow the training recipe of the Resnet-50 model in open clip, thus both the image encoder and text encoder are trained from scratch on synthetic noisy YFCC15M. For CLIP fine-tuning (i.e., our noisy model tuning), we fixed all the encoders.