Noise Separation guided Candidate Label Reconstruction for Noisy Partial Label Learning

W1:

Due to the relative simplicity of the CIFAR10 dataset on setting $\eta$=0.5, $\gamma$=0.3 and the benefit of our proposed ECK metric, a well-pretrained single-threshold strategy can still achieve satisfactory performance. As shown in the table below, the gap between Ours and Ours v2 widens as the situation becomes more challenging, such as with more difficult dataset (CIFAR100) or higher noise rates. This demonstrates the effectiveness of our dual-threshold strategy, particularly in handling more complex scenarios.

W2:

ECK is used to measure the inconsistency error between KNN-based pseudo-labels and CLS-based pseudo-labels. It serves as a tool to distinguish normal samples from noisy samples. As you suggested, we conducted experiments in which the roles of KNN-based pseudo-labels and CLS-based pseudo-labels were interchanged during the calculation of ECK. The results indicate that Reverse ECK outperforms the current SOTA method ALIM, highlighting the effectiveness of our proposed approach. However, the original method achieves better performance than the reverse one.

W3:

We appreciate your observation regarding this typo. It has been corrected in the latest version of the paper, and we will conduct a thorough review to avoid the typos in the manuscript.

W4:

Thank you for your suggestions. We conducted extensive experiments across a wide range of datasets. Our method was first evaluated on two benchmark datasets, CIFAR10 and CIFAR100, with various noise and ambiguity levels. Additionally, we tested on three fine-grained datasets (CIFAR100H, CUB200, Flower) and validated our approach on two real-world crowdsourced datasets (Treeversity and Benthic). Therefore, we believe our experimental evaluation is both comprehensive and diverse.

However, as you suggest, we further conduct experiments on a large WebVison dataset which contains ∼2.4 million images based on queries generated from the 1,000 ImageNet classes. As this dataset was crawled from the web, its training set inherently contains noise. Thus, generating the NPLL dataset requires only the specification of the noise (ambiguity) rate. It is important to note that the WebVision test set is noise-free. To expedite training, we utilized samples from the first 50 classes for both training and testing. The experimental results of test accuracies of each method presented in the table below further validate the effectiveness of our proposed method on larger dataset.

Dear reviewer RE2k,

Thanks again for your time and efforts in reviewing this paper and the valuable comments on improving its quality. As the reviewer-author discussion deadline approaches, please take a few minutes to read the rebuttal. If you have further concerns, we are happy to provide more explanations. Thanks.

Regards from the authors.

Thanks for the authors' response. For Q2, I am not sure whether the authors misunderstand my quesiton. I mean replace the $p_i$ in Eq.(2) with the $q_i$ computed by Eq.(1).

Moreover, there are two pseudo labels: CLS-based pseudo label and KNN-based pseudo-label, which of them can be the better one in your setting.

W1 & W2:

W1 and W2 are duplicated.

We would like to address a misunderstanding regarding our method. Our approach does not involve training a new classifier in each epoch but is instead a plug-in strategy designed to be integrated into any existing PLL model for defeating noise. In each epoch, our method refines the dataset before introducing it to the PLL model. This refinement process requires only the sample features and predicted probability vectors stored from the previous epoch of the PLL model, only resulting in an additional computational complexity for KNN search and ranking. Besides, the code of our method has been submitted to the Supplementary Material. You could check and run the code to get more implementation details.

By utilizing the Faiss library for high-speed KNN search on GPUs, the additional computational overhead is minimized. To validate our claim, we integrated our method into Papi (Xia et al., 2023) and conducted experiments on CIFAR100 with an NVIDIA RTX 3090 GPU. The results showed that the time per epoch increases only 8.67%, supporting the efficiency and practicality of our approach. Although the integration of our method introduces a slight computational time overhead, it yields substantial performance improvements. As presented in Table 2, on the CIFAR-100 dataset ($\eta=0.1$), our approach outperforms the original method (PaPi) with accuracy gains of 12.14%, 16.25%, and 22.50% under noise rates of 0.2, 0.3, and 0.4, respectively.

W3:

Higher noise rates intensify the difficulty of the NPLL problems, as evidenced by a significant performance decline across all methods on Table 1 of the manuscript. However, at higher noise levels, the performance gaps between our method and other methods become larger. For example, on CIFAR100 with $\eta=0.1$, our method improves the best compared method by 4.71%, 8.15% and 11.57% on three different noise levels $\gamma=0.2,0.3$ and $0.4$ respectively. The above analysis proves that our method is more reliable than the current methods under more noisy samples.

Q:

We have never made the claim that "unreliable samples become increasingly unreliable." On the contrary, we believe that as the model undergoes training, its capability improves, leading to a gradual reduction in unreliable samples. This enables more reliable distinctions between normal and noisy samples (See Fig. 3 and Fig. 4(a)).

In the early stages of training, the model lacks sufficient capacity to accurately classify samples as normal or noisy due to its limited training. To address this, our approach adopts a dual-threshold sample separation strategy, which uses two thresholds to categorize samples into high-confidence normal samples, high-confidence noisy samples, and uncertain samples. As the training progresses and model's capability improves, the two thresholds converge gradually, ultimately achieving accurate separation.

Figure 3 of the paper illustrates the ECK distribution for normal and noisy samples during training. It shows that as the number of training epochs increases, the model becomes progressively better at distinguishing between normal and noisy samples. Additionally, the number of high-confidence samples (including both high-confidence normal samples and high-confidence noisy samples) steadily grows throughout the training process, eventually leading to the separation of all samples.

Dear reviewer Xu7T,

Thanks again for your time and efforts in reviewing this paper and the valuable comments on improving its quality. As the reviewer-author discussion deadline approaches, please take a few minutes to read the rebuttal. If you have further concerns, we are happy to provide more explanations. Thanks.

Regards from the authors.

Dear reviewer Xu7T,

Thank you once again for your time and effort in reviewing our paper and providing valuable feedback to enhance its quality. With the PDF submission deadline approaching tomorrow, we kindly request your review of our rebuttal. If you have any further concerns, we would be pleased to provide more detailed explanations.

Regards from the authors.

Dear Reviewer Xu7T,

As the Reviewer-Author discussion phase is drawing to a close, we kindly ask you to review our revisions and responses once more and reconsider your rating. All the other reviewers' concerns have been resolved, and they all gave this paper a positive score. We eagerly anticipate your feedback. Thank you.

Best regards,

The Authors

W1 & Q2:

The main contribution of the BABS (Lv et al., 2024) lies in its exploration of the average-based strategy (ABS) in PLL and provide a theoretical robustness analysis for the average PLL losses under ABS. However, BABS have already been outperformed by the method UPLLRS (Shi et al., 2023) that we have outperformed.

But, as suggested, we compare with it in the Table below. In BABS, it was demonstrated that bounded loss functions, such as Mean Absolute Error (MAE) and Mean Square Error (MSE), exhibit robustness to NPLL. Among these loss functions, MAE achieves the best experimental performance in this work. Therefore, MAE is adopted as the baseline in the experiment. The results show our method outperforms BABS significantly. And the perfermance gap extends under more difficult situations.

Q1:

The accuracy of the CLS pseudo-label is usually lower than the KNN-based pseudo labels especially in the high noisy rate situations. Specifically, when the sample is a noisy sample (i.e., the ground-truth label is not in the CLS), the CLS pseudo-label is completely incorrect. In contrast, KNN pseudo-labels are constructed based on feature space that not directly affected by the noise CLS. Fig. 4(b) of the paper proves that the reconstructed CLS based on the KNN-pseudo labels $\boldsymbol{q_i}$ in Eq. (6) maintains an almost 100% ground-truth hit rate during model training, suggesting the rationality of the current approach.

Besides, as suggested, we supplemented the following experiments: 1. KNN-based: the CLS reconstruction method used in our paper; 2. CLS-based: using CLS-based pseudo-labels for CLS reconstruction; 3. Fusion: combining CLS-based pseudo-labels and KNN-based pseudo-labels for CLS reconstruction. The following experimental results further validate the correctness of our approach.

Although the accuracy of CLS-based pseudo-label is not high, our objective in introducing CLS-based pseudo-label is to examine whether the sample is a noisy sample by comparing it with KNN-based pseudo-label. For noisy samples, there will be a significant difference between CLS pseudo-label and KNN pseudo-label, resulting in a high ECK value. Conversely, if the ECK value is low, it indicates that it is a normal sample. As illustrated in Fig. 3 of the paper, the ECK values for noisy samples are significantly higher than those for normal samples.

Dear reviewer 1LBQ,

Thanks again for your time and efforts in reviewing this paper and the valuable comments on improving its quality. As the reviewer-author discussion deadline approaches, please take a few minutes to read the rebuttal. If you have further concerns, we are happy to provide more explanations. Thanks.

Regards from the authors.

Dear reviewer 1LBQ,

Thank you once again for your time and effort in reviewing our paper and providing valuable feedback to enhance its quality. With the PDF submission deadline approaching tomorrow, we kindly request your review of our rebuttal. If you have any further concerns, we would be pleased to provide more detailed explanations.

Regards from the authors.

Thanks for the authors' response. I will keep my score.

W1:

The issue you mentioned is a common challenge for the noisy partial label learning (NPLL). But our proposed method is designed to relieve this challenge. Specifically, we present a candidate label set (CLS) reconstruction approach aimed at generating shorter and more precise CLS. This approach facilitates the model's ability to identify ground-truth labels, thereby improving performance. As depicted in Fig. 4(b) of the paper, our method empirically reduces the size of the reconstructed CLS consistently over the training iterations (represented by the green curve). Accordingly, on the dataset with a larger label space, our method achieves a substantially greater performance improvement compared to other baseline methods e.g., on CIFAR100 ($\eta=0.1$), our method improves the best compared method by 4.71%, 8.15% and 11.57% on three different noise levels $\gamma=0.2,0.3$ and $0.4$ respectively, while on CIFAR10 ($\eta=0.5$), our method performs 2.48%, 3.83% and 7.04% better than the best compared method (Table 1 of the paper).

W3:

We appreciate your valuable comments. Following your suggestion, we have incorporated a hyper-parameter sensitivity analysis into the appendix (A.8) and updated the submitted paper accordingly. Specifically, we have performed sensitivity analysis on four hyper-parameters: the warm-up epoch ($e_w$), the rate of percentile change between $r_l$ and $r_u$ ($\lambda$), the trade-off of two objectives on CLS Reconstruction ($\beta$), and the KNN parameter ($K$). Results show that $\beta$ and $K$ are stable, with optimal values at $\beta = 1.5/2.0$ and $K = 7/5$, achieving peak accuracies of approximately 95.5%/80.4% for CIFAR-10/100. $\lambda$ performs best at $0.4$ but is relatively sensitive to noise levels in the dataset, requiring careful adjustment. $e_w$ is crucial for training stability, as insufficient warm-up leads to poor pseudo-labeling, while excessive warm-up causes overfitting to noisy labels in the CLS; the optimal value is $150$. Overall, $\beta$ and $K$ are robust, whereas $\lambda$ and $e_w$ need carefully tuning. The detailed analyses and the experimental reulsts can be found in the Appendix A.8 of the paper.

Q1:

In PLL, each sample has a CLS. And in NPLL, each sample remains associated with a CLS, as in PLL. So the CLS of all samples, including both normal samples and noisy ones are accessible. However, different with PLL, some samples' ground-truth labels locate outside the CLS in NPLL, and the these samples (denoted as noisy samples) are mixed with normal ones. The introduction of CLS-based pseudo label computation in Eq. (2) aims to facilitate the distinction between normal samples and noisy samples in subsequent processing stages. Therefore, in Eq. (2), CLS-based pseudo labels are computed for all samples including both the normal ones and noisy ones. In the following subsections, we reconstruct shorter CLS for all samples to replace the initial ones to get the better performance.

Q2:

Our empirical observations indicate that our method achieves consistently high performance with little variation across this interval when the parameter $K$ is set within the range of 5 to 10 on all datasets. Therefore, for the sake of simplicity and consistency, we fixed $K=5$ in all experiments unless the hyper-parameter sensitivity analysis experiments in Section Appendix A.8.

Q3:

Figure 3 of the paper presents the ECK distribution for normal and noisy samples during model training on CIFAR10. The results show that noisy samples (i.e., those with ground-truth labels outside the candidate label set) exhibit larger ECK values compared to normal samples. Furthermore, as training progresses, the gap between the ECK distributions of these two groups of samples becomes more pronounced. This observation supports our claim that the proposed ECK metric effectively distinguishes noisy samples from normal ones. Similar findings are observed on another dataset (CIFAR100), as shown in Figure 6. Additionally, we performed comparisons of separation accuracy with existing NPLL methods in Appendix A.6, demonstrating that the proposed ECK metric is more effective in distinguishing noisy samples from normal ones compared with other NPLL approaches.

Q4:

Yes, this operation will be beneficial to the performance. The detailed explanations and ablation experiments are as follows.

Our dual-threshold method categorizes samples into reliable samples (including reliable normal samples and reliable noisy samples) and uncertain samples. For reliable samples, CLS reconstruction is performed as described in Eqs. (6)–(9). For uncertain samples, their type as either normal sample or noisy sample cannot be well determined. Directly training on those uncertain samples with their original CLS would risk overfitting to label noise, leading to performance degradation. Conversely, discarding uncertain samples entirely would reduce the training sample size, which also negatively impacts performance.

To address the above dilemma, we propose Eq. (10), which tries to refine the CLS of the uncertain samples by swapping the label with the lowest confidence in the CLS and the label with the highest confidence in the non-CLS. This correction mitigates hidden noise in the samples to some extent without increasing the CLS length. It is worth noting that the number of uncertain samples is limited and decreases over iterations (see Figure 3 of the paper), so their impact on the model is also relatively limited. As a summary, the correction of noise is primarily achieved through CLS reconstruction by Eqs. (6)-(9), while the main role of Eq. (10) is to prevent the model from overfitting to the noise within these uncertain samples.

We validated the effectiveness of Eq. (10) through experiments on CIFAR100 ($\eta = 0.05, \gamma = 0.3$) and CIFAR100 ($\eta = 0.05, \gamma = 0.4$), with the results presented in the table below. Here, Ours represents our proposed method, Ours Original uses the original CLS for uncertain samples during training, and Ours Discard discards uncertain samples entirely. The results show that our method consistently achieves the highest performance. Notably, under higher noise rates, the gap between Ours and Ours Original widens, further highlighting the effectiveness of our approach.

Q5:

The first inequality of Eq. (18) is

Since $\mathcal{L}(f(\boldsymbol{x}), y)$ is upper-bounded by $M$ ($\mathcal{L}(f(\boldsymbol{x}), y) \le M$), we can deduce

Define $m = \sum _ {i=1}^{n} \mathbb{I}(\boldsymbol{y} _ i \in Y _ i)$, which implies the noise rate $\epsilon = \frac{n - m}{n}$. Define the function $f(|Y|) = \frac{|Y| - 1}{|Y|}$, which is concave. By Jensen's inequality and the definition of $\alpha$ ($\alpha=\frac{1}{m} \sum _ {j=1}^{m} |Y _ j|$), we have

Combing the above two inequalities, we have

At this point, the second line of Eq. (18) is complete.

Q6:

Thank you very much for pointing out the typo, which have been corrected in the revised version.

Dear reviewer xLDw,

Thanks again for your time and efforts in reviewing this paper and the valuable comments on improving its quality. As the reviewer-author discussion deadline approaches, please take a few minutes to read the rebuttal. If you have further concerns, we are happy to provide more explanations. Thanks.

Regards from the authors.

Q1(a):

Thank you for highlighting the undefined notation $f _ y(x)$ in Line 125. You are correct that this term is intended to represent the predicted probability of the $y$-th class for $x$. We have explicitly defined $f _ y(x)$ upon its first appearance in the updated paper (PDF). This clarification will ensure that readers can understand the notation without ambiguity.

Q1(b):

Yes, as pointed out by you, the Rademacher complexity does not necessarily converge to zero. Theorem 1 aims to demonstrate that as the sample size increases, along with a reduction in noise levels and mean CLS size, the gap between the generalization error $R(\hat{f})$ of the model $\hat{f}$ trained on the NPLL dataset and the generalization error $R( f ^ { * } )$ of the optimal model $f ^ { * }$ diminishes, i.e., the generalization performance gap between $\hat{f}$ and $f^{*}$ decreases. The term "converge" might introduce ambiguity in the context. Therefore, we revised the original sentence as "As $n \to \infty$, $\epsilon \to 0$ and $\alpha \to 1$, Theorem 1 shows that the generalized error bound will be reduced, and the empirical risk minimizer $\hat{f}$ will get closer to the true risk minimizer $f^{*}$." and updated the manuscript accordingly.

Q2(a):

ECK is calculated based on the difference (cross-entropy) between CLS-based pseudo-labels ($\boldsymbol{\widetilde{q} _ i}$) and KNN-based pseudo-labels ($\boldsymbol{q _ i}$). For normal samples, since the ground-truth label locates in the CLS, as the model is training, both the KNN-based pseudo-labels and CLS-based pseudo-labels will increasingly approximate the ground truth. Therefore, according to the properties of cross-entropy, a smaller ECK value will be produced. For noisy samples, because the original CLS does not include the ground truth label, while KNN-based pseudo-labels may contain the ground-truth label if the model is well-trained on reconstructed CLS. Consequently, the discrepancy between KNN-based pseudo-labels and CLS-based pseudo-labels (based on the original CLS) is large, resulting in a larger ECK value.

Figure 3 of the paper illustrates the ECK distributions for normal and noisy samples during model training, showing that noisy samples exhibit larger ECK values compared to normal samples. Similar results are observed on another dataset, as presented in Figure 6. Moreover, comparison results in Appendix A.6 demonstrate that the proposed ECK metric outperforms existing NPLL methods in distinguishing noisy samples from normal ones, providing superior separation accuracy.

Q2(b):

We have incorporated a hyper-parameters sensitivity analysis including warm-up epoch into the Appendix A.8. From the experimental results, we observe that the warm-up epoch ($e _ w$) plays a critical role in training stability. Insufficient pretraining with a small $e _ w$ can result in poor feature discrimination, which hinders the generation of reliable pseudo-labels. Conversely, an excessively large $e _ w$ may cause overfitting to noisy samples, reducing the model's ability to correct errors during later training stages. In addition, Table below compares the impact of $e _ w$ under different noise levels on the CIFAR-100 dataset. The results show that as the noise rate increases, the model becomes more sensitive to the choice of $e _ w$, indicating the necessity of careful tuning.

Higher noise rates exacerbate the challenges of the NPLL problem, as evidenced by a significant performance decline across all methods on Table 1 of the manuscript. However, at higher noise levels, the performance gaps between our method and other methods become larger. For example, on CIFAR100 with $\eta=0.1$, our method improves the best compared method by 4.71%, 8.15% and 11.57% on three different noise levels $\gamma=0.2,0.3$ and $0.4$ respectively. The reason is that the proposed sample separation approach can correctly identity the noisy samples, and accordingly relieve the challenges of higher noise rates significantly.

Dear reviewer B6bW,

Thanks again for your time and efforts in reviewing this paper and the valuable comments on improving its quality. As the reviewer-author discussion deadline approaches, please take a few minutes to read the rebuttal. If you have further concerns, we are happy to provide more explanations. Thanks.

Regards from the authors.