Cleaning label noise with vision-language models

Thanks for the insightful reviews and acknowledgement about the experiments and presentation. We are happy to answer your raised questions below:

*Q1.1.This paper uses the CLIP pre-trained model, I think this is unfair for previous works without pre-trained model. *

We understand the concerns of the reviewer but we chose the specific setting precisely so as to perform fair comparisons: as it stands, all our comparisons are using the same encoders (e.g. PreResNet18) as reported in the original papers. We note that , in several cases, CLIP-based encoders may be heavy and not desired to be used. In this condition, sample selection enables offline and efficient usage of CLIP model and ensures a fair comparison for all methods.

Our main motivation is to introduce the benefits of using vision-language models into the “Learning with Noisy Labels” community -- we believe that this is a significant contribution. We show that with a very simple scheme we achieve SOTA. There are indeed many other ways that VL models could be used in this domain, we simply use CLIP for sample selection to avoid its direct impact on the in-training model to further ensure a fair comparison. We hope that other researchers in the field of LNL will explore using VL methods in their work. Please refer to our reply to Q2 of Reviewer s2nG for more discussions and results.

Q1.2.Combining previous methods with training from CLIP pre-trained model other than training from scratch should also be compared.

Integrating CLIP with existing techniques for learning with noisy labels is an interesting research direction that we intend to explore in the future. Preliminary experiments demonstrate that a pretrained clip backbone does helps. In this paper, however, we focus on utilising CLIP as a sample selection method that does not require any adaptation to the training of the classifier, and achieve SOTA/competitive results. We believe that this is a sufficient contribution.

Q2.Equations (2) misses ).

We apologize for the typo, we are fixing this along with other changes. We will upload the updated verison later.

If the reviewer has any additional questions or anything else to discuss, we are more than happy to engage further in the conversation.

[1] SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise.

Thanks for the insightful reviews and acknowledgement about the method and presentation. We are happy to answer your raised questions below:

Q1.1.My major concern is the potential unfair comparisons. The notable performance improvements shown in Tables 2-4 could be attributed to CLIP's superior representation learning capabilities. A fairer comparison could involve replacing the baselines' backbone with CLIP's visual encoder.

We understand the reviewers' concerns, but we chose the specific settings precisely to allow for fair comparisons: as it stands, all our comparisons use the same backboneas reported in the original paper. Our main motivation is to bring the benefits of using visual language models to the "Learning with Noisy Labels" community -- we believe that this is a significant contribution. We show that with a very simple solution we achieve SOTA. There are indeed many other ways that VL models could be used in this domain, we simply use CLIP for sample selection to avoid its direct impact on the in-training model to further ensure a fair comparison. And we hope that other researchers will explore them in their work. See our response to reviewer s2nG Q2 for more discussion and results.

Q1.2.A fairer comparison could involve replacing the baselines' backbone with CLIP's visual encoder.

Please refer to our response to reviewer Vnif Q1.2.

Q1.3. Furthermore, Table 3 lacks comparison results with recent works focused on instance-dependent noise from 2022-2023.

We will incorporate more recent results in Table 3 in the updated verision. Due to time limitation, we present below an initial version in Table R2 with one more recent work for your perusal:

Table R2. More results on instance-dependent noise from [2]

We are happy to include more discussions if the reviewer is aware of any other related works reported on the same bencmarks. There are also other works proposing different models for instance-dependent noise such as [2]. We here report extra results in Table R3 with noisy labels provided by [3] (https://github.com/pxiangwu/PLC/tree/master/cifar/noisy_labels). According to Tables R2 and R3, we can verify that at various noise ratios and noise types, significant improvements are obtained by simply introducing CLIP-based sample selection.

Table R3. Results on instance-dependent noise from [3]

Q2.Discrepancies between the CLIP's zero-shot results on CIFAR in Table 1 and the original paper need clarification.

Admittedly, reproducing the reported results from the original CLIP paper is actually challenging, as acknowledged in the discussion here: https://github.com/openai/CLIP/issues/164. The reported numbers in our paper were obtained under consistent experimental settings to ensure a fair and reliable comparison.

We appreciate your prompt reply. We would like to further clarify your concerns below:

Q7: Backbone Replacement with CLIP's Visual Encoder.

To finetune the CLIP visual encoder is another available style utilizing a vision-language model for LNL. Instead, we use offline sample selection, specifically because:

• Acknowledgement: sample selection is the dominant paradigm for LNL, by utilizing CLIP for sample selection our method can be seamlessly utilized with existing methods. For example, simply adding an extra warmup stage before the original training frmework with the selected samples from CLIP.

• Effectiveness: Viewing CLIP as a 'black-box foundation model', considering the possible computational cost/deployment issues, etc, the most direct and convenient way to utilize it would be the offline one.

We expect our method to motivate the usage of CLIP in the LNL community since CLIP with only naive/offline sample selection shows competitive results. We hope that other researchers in the field of LNL will explore other styles using vision-language methods in their work.

Fine-tuning experiment: Regarding the suggestion to replace the model with CLIP's pretrained model, we here report fine-tuning results on SSR[1], as mentioned by Reviewer Vnif. Specifically, we fine-tune the CLIP pre-trained model with VIT-B/32 backbone, using its visual encoder with an additional linear layer classifier. We initially froze the encoder, train the classifier for one epoch with LR=0.02, and subsequently train the whole model with LR=0.001 and LR_scheduler=CosineAnnealing for 100 epochs, with a weight decay of 0.01. As a comparison, we use the same structure but randomized the weights of the encoder, train for 100 epochs with LR=0.001 and LR_scheduler=CosineAnnealing, with a weight decay of 0.01. Initial results are presented below in Table R4:

Table R4. SSR with CLIP

As expected, the pre-trained CLIP encoder provides effective improvement, further demonstrating the potential of the CLIP model for LNL. Also, please note that fine-tuning CLIP is not trivial, as described in the following GitHub issues. Through better hyperparameter settings, we believe there is still room for further progress.

• https://github.com/openai/CLIP/issues/83
• https://github.com/openai/CLIP/issues/150
• https://github.com/openai/CLIP/issues/362

---

Q8: Paper's Clarity.

Regarding the presentation of the paper, we are currently making revisions to the manuscript based on the feedback from all reviewers. We would like to convey that the majority of the changes are straightforward and actionable.

We hope that this further clarification addresses your concerns. If you have any additional questions, we are open to further discussion.

[1] SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise, BMVC2022.

We appreciate your prompt reply and willingness to engage in further discussion. However, we would like to respectfully reiterate that comparing the online usage of large-scale pretrained CLIP models to offline sample selection of CLIP falls outside the scope of this work - as discussed in the paper, the best style for CLIP in LNL is still underexplored. Besides, It's a well-established fact that directly finetuning large-scale pretrained models is highly possible to yield superior performance compared to offline usage. Our primary goal is to introduce CLIP to the LNL community, and we believe this comparison is unnecessary and distracts from our focus. We kindly refer the reviewer's attention to the <Additional comments about the fairness and contribution of the proposed method> response section for a more in-depth explanation of our approach's significance. Thanks for your time and consideration.

Specifically, regarding the numbers in Table R4, we can actually never make a comparison. For example, on CIFAR dataset, CLIP model applies VIT-B/32 with 224x224 px image input, while all current works utilize PreResNet18 with 32x32 px input. This is why we insist a offline usage of CLIP model to avoid effects of these extra factors in the original paper.

Thanks for the insightful reviews and suggestions. We are sorry about the confusion caused by the presentation and we appreciate that the reviewer checked the appendix to find some important points-- we will be happy to move material between the main paper and the appendix as the reviewer suggests. We are happy to provide further clarifications below:

Q1: Eq1: Sample selection mechanism takes as input the predicted probability and the label? Please clarify.

Yes, as formalized in sec 3.1, current sample selection methods usually rely on an estimated probability vector and noisy labels. Intuitively, a ‘distance’ or ‘consistency’ is calculated between the estimated probability and the annotated label, to measure the ‘cleaness’ of the label, such as the per-sample losses.

Q2: Typo: Sec 3.2: “we consistent the notations for CLIP’s training”

We apologize for the typos here, we will fix it as ‘We make the notations consistent for CLIP's training’.

Q3: Clarify Sec 4.1 “clean test set of in-question noisy datasets”

We apologize for any confusion. To clarify, when we mention ‘the clean test set of in-question noisy datasets’, we mean the clean test set corresponding to the noisy train set. We will address and rectify this point in the updated version of the paper.

Q4: Appendix F: "label noise normally does not affect the sample itself": Label noise can be highly sample dependent, so I am unsure what the authors mean by this statement.

Following previous works in LNL community, we assume the causal mechanism between sample $x$ and label $y$ as $x \rightarrow y$. Thus we assume the conditional probability ($P(y|x)$ - noisy labelling) will not affect the sample coefficient prior ($P(x)$).

Q5: Estimate P˜(yi|xi) with CLIP zero-shot classifier: Re-formulating the CLIP loss, including the fact that sampling at a prompt level might yield a better ZS estimate. None of this is new, but it reads as though the authors are claiming this formulation as new. It will help to state that this is a reformulation of the standard CLIP ZS process, the only addition being a different prompt template, basically just Eq. 4. In addition, multiple prompt generation uses another language model that has its own biases which are conveniently not accounted for in the main text and case into the appendix.

Sorry for the confusion here. We do not claim the CLIP zero-shot classification as our contribution, rather we simply want to present the multi-feature prompting in our framework. We will improve the presentation in the updated version.

Q6: Using CLIP is suboptimal in one key manner since we dont have access to the train set, we are unsure of the biases existing in the CLIP model.

Indeed, biases, errors and domain shifts are risks inherent when using any pre-trained model. With a very large train set, we expect the CLIP model can cover a wide range of domains. In comparison to ordinary vision-only pre-trained models, CLIP still possesses its unique advantages due to its language modality and zero-shot classification capability, offering sample selection that is not influenced by noisy labels.

Q7: Section 4.2: “synthetic symmetric/asymmetric noise.” What is this noise model?

We follow the widely-acknowledged definitions of symmetric and asymmetric noise in the LNL community. In particular, symmetric noise is generated through random label flipping, while asymmetric noise is induced by specific transitions between semantically similar classes, such as 'horse' ↔ 'deer' for the CIFAR-10 dataset.

If the reviewer has any additional questions or anything else to discuss, we are more than happy to engage further in the conversation.

Thanks for the insightful reviews and acknowledgement of our exploration. We are happy to answer your raised questions below:

Q1.The second method to estimate the \tilde{p}{y|x} seems similar to the estimation of p{y|x} with noisy labels. Since the classifier is learned with noisy data, how can it be used to estimate the clean probability? Authors should provide more explanation for this problem.

We appreciate your concerns. As elucidated in Theorem 2, the second method is affected by label noise since we train a new classifier based on it. This is the motivation behind presenting two distinct methods for estimating $P(y|x)$ using CLIP. We believe that CLIP, with its unique language modality compared to ordinary pre-trained visual models, holds a significant advantage, particularly because the first method is entirely free from noise.

Q2.The results are inferior to many state-of-the-art methods, such as Unicon, PES, etc.

We appreciate your concerns and would like to emphasize that our method is not mutually exclusive but can be used in conjunction with existing techniques. For example, we can utilize CLIPSelector seamlessly with any existing methods including the mentioned UNICON and PES, by adding an extra warmup stage with only the selected samples from CLIPSelector. Due to time limitation, as an illustration, we present results after incorporating model co-training (one of the validated effective and simple technique) with our method (by exchanging the selected samples between two models) on CIFAR100 and Animal-10N datasets (Table R1), further demonstrating that our method has great potential along with existing techniques and it is comparable with existing works.

We will include above discussions in the updated version.If the reviewer has any additional questions or anything else to discuss, we are more than happy to engage further in the conversation.

[1] UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning, CVPR2022.

[2] Understanding and Improving Early Stopping for Learning with Noisy Labels, NeurIPS2021.