Realistic Evaluation of Deep Partial-Label Learning Algorithms

First of all, we are very grateful for your help and time in reviewing our submission. Below are the responses to your questions and comments.

Q1: Could the authors comment on the differences and similarities between their work and [1,2]? Both papers appear to address benchmarking in the PLL setting and should be discussed to provide a clearer context.

A1: The two papers do not work on PLL, although their data can also be annotated with multiple labels. The similarity is that we all study datasets annotated with multiple labels. The differences between our work and theirs are:

• The problems studied are different. First, the task of [1] is to predict soft labels given a dataset with crowdsourced soft labels. [2] works on noisy label learning, which aims to predict a single label given a dataset with label noise, and they are mainly single-label, while some can filter a single noisy label from multiple noisy labels. Our task (PLL) is to predict a single label given a dataset where each example has multiple candidate labels, among which only one label is correct. Therefore, the problem settings are different and we are the first benchmark for PLL.

• The evaluation goal is different. [1] aims to evaluate a labeling model, which is a model trained to annotate labels. [2] mainly works on noisy label detection, which aims to find noisy label examples from the dataset. Our paper works on algorithm evaluation, which aims to evaluate the classification performance of different algorithms.

• Our work also provides new model selection criteria with theoretical guarantees, which fills a gap in the PLL literature.

Therefore, there are clear differences between these two papers and ours. We have cited them in the revised version of the paper.

Q2: Could the authors provide experimental evidence showing that the proposed criteria lead to better performance on the test dataset?

A2: The advantage is twofold. First, for the hyperparameter selection, we found that the test performance for our selected ones is better than the default hyperparameter in most cases. This is because we also add the default hyperparameter configuration to the selection pool, but always find that other hyperparameter settings are better than it. Second, early stopping helps us get better results. For example, from Tables 1 to 4, we can see that OA & ES is always better than OA. This conclusion is also true for CR and AA. Therefore, we can use our proposed criteria for early stopping to achieve better performance.

Q3: Would it be possible for the authors to include a numerical comparison showing how often each criterion achieves the best performance?

A3: In Table 1, CR wins 13 times, AA wins 3 times and OA wins 10 times. Similar results can also be obtained from other tables. We can see that CR can achieve the best performance in most cases. However, for specific algorithms and cases, we should specifically determine the best criterion, and the three criteria can be complementary to achieve better classification results.

Q4: Could the authors provide further explanation of $p(\boldsymbol{x},y)$ and $p(S|\boldsymbol{x}, y)$?

A4: $p(\boldsymbol{x},y)$ denotes the joint distribution of the instance $\boldsymbol{x}$ and the label $y$. $p(S|\boldsymbol{x}, y)$ denotes the conditional probability distribution of $S$ given $(\boldsymbol{x},y)$. We have added descriptions after Eq. (4) and the calculation of $p(S|\boldsymbol{x}, y)$ can be found in Equations (1) and (2) for example as well.

Q5: I want to bring two particular papers [3,4] into the author's sight.

A5: Thanks for the helpful reference. Both papers work on similar problem settings and we have cited them in the revised version.

Thanks for the response. I have a following up question regarding Q1. I originally thought [1,2] to be similar to this paper, and the authors clarified it. After reading the response, I wonder if [3] is actually more similar. Essentially, I want to understand how PLL differs from the previous multi-label learning. Thanks.

Dear Reviewer GD1c,

We are very grateful that you have kindly raised your score on our submission. We sincerely appreciate the time and effort you have invested in reviewing our submission. Your comments and suggestions help us improve the quality of our work. Thank you again for your time and help!

Best regards,

ICLR 2025 Conference Submission5419 Authors

First of all, we would like to express our sincere gratitude for your efforts and time in reviewing our submission. Below are the responses to your questions and comments.

Q1: Confusing terminologies: model selection vs hyper-parameter tuning.

A1: We agree that it is helpful to express more carefully in this paper that the proposed criteria are mainly for hyperparameter selection. We have added it in the paper. Besides, our wording mainly refers to DomainBed [1], which also proposes some model selection criteria for hyperparameter tuning. Although we mainly use them to select hyperparameters, we can also use them to select appropriate models from a model pool. For example, we can determine which of ResNet and DenseNet is better based on our criteria, since the deeper one (DenseNet) may not always be a better choice according to our experimental results. We believe that our proposed criteria can be used in broader aspects than just hyperparameter selection.

Q2: Confusing toy example in Figures 1a and 1b.

A2: We apologize for our unclear presentation. The lighter and darker colors indicate different ways to use a clean validation set for a given algorithm. The lighter colors indicate that we use it only to tune the hyperparameters, and the darker colors indicate that we add it to the training set without carefully tuning the hyperparameters. We can see that using it for training can often lead to better results for many PLL algorithms, showing that current experimental settings of PLL may not be appropriate. The floating text means used algorithms, we float them to align them to the two right panels since the lengths of the algorithm names are very long and can take up a lot of space. We have revised the captions and figures to make them clearer.

Q3: Complicating notations.

A3: We agree with you and have revised $\mathbb{E}\left[{\rm ACC}(f)\right]$ to ${\rm ACC}(f)$.

Q4: Proof of Proposition 1.

A4: First, we have revised Lemma 1 to Definition 2 as you suggested. Then we explain the proof of Proposition 1 in Appendix A.1. The key idea of it is to unfold the difference as an expectation for a term for each data point and then upper bound the values by our introduced definitions. The first equation is according to the definitions there. The second is natural if the wrong prediction is in the candidate label set. The third equation can be obtained by traversing as said in the paper. The fourth equation is by the definition of expectations. The fifth equation is by the conditional probability equation, and the last inequality is by the definition of Ambiguity Degree.

Q5: Unclear distinction between two synthetic datasets.

A5: Sorry for the error. We have corrected the sentence. The correct phrase should be: The second is PLCIFAR10-Vaguest, which assigns each example the largest candidate label set from the annotators. The two datasets are different.

Reference:

[1] In Search of Lost Domain Generalization, ICLR 2021.

First of all, we would like to thank you for your time and effort in reviewing our paper. We are very grateful for your helpful and high quality comments and suggestions. Below are the answers to your questions and comments.

Q1:OA should be presented as a baseline, not as a novel contribution.

A1: We agree with your comment. We have modified the paper to show that OA is not contribution.

Q2: It is unclear how ES is used in the experiments. Is ES used in the case of CR and AA?

A2: Following DomainBed [1], we used ES for CR and AA. The detailed descriptions can be found in Section 5.3. Specifically, we recorded the values of CR and AA on a validation set and the test accuracy for all iterations. We then selected the iteration with the highest value of a criterion and returned the corresponding test accuracy. Notably, the selected iteration may be in the middle stage of training, which can have similar effects to ES.

In real applications, we can also check the values of these criteria and perform ES when we find that the value of these criteria starts to decrease.

Q3: Difference between Aggregate and Vaguest not clear.

A3: Sorry for the error. We have corrected the sentence. The correct phrase should be: The second is PLCIFAR10-Vaguest, which assigns each example the largest candidate label set from the annotators.

Q4: The subfloats in Figure 2 are not aligned at the top.

A4: We have revised the figure as you suggested.

Q5: I was wondering if there is access to the identities of the annotators and how the authors ensured their privacy?

A5: Annotators' identities are not accessible through the platform interface. Their privacy is strictly protected by the Amazon MTurk platform.

Q6: Can you please provide rational on how you selected the payment rate?

A6: The payment rate was chosen based on our funding and the characteristics of the problem setting. We first experimented with this payment rate on a small amount of data and found that the noise rate was similar to previous work using different payment rates [2]. For example, on the CIFAR-10 dataset, CIFAR-10N-random has a noise rate of 18%, while our PLCIFAR10-Vaguest dataset has a noise rate of 17.56%. Therefore, we thought that the annotation quality of our payment rate might be comparable to previous work. In addition, we found that the tasks were completed quickly in a short period of time, so we ultimately set this payment rate. Thank you for providing the references and we have cited them in the paper.

Reference:

[1] In Search of Lost Domain Generalization, ICLR 2021.

[2] Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations, ICLR 2022.

Thanks for your detailed and thoughtful responses. The explanations have effectively addressed all the raised points and resolved my concerns.

First of all, we greatly appreciate your time and efforts in reviewing our paper. We am encouraged and thankful that you agreed with the contributions of the paper. Below are the responses to your questions and comments.

Q1: What changes do you expect to see on larger datasets and with more complex deep neural networks?

A1: Thank you for your valuable question. Using larger datasets with more complex patterns is very promising for partial-label learning (PLL). However, the performance of many PLL methods may decrease because they have made simple assumptions about the data generation process that may not be compatible with large complex datasets. Therefore, it is promising to evaluate PLL methods with larger datasets in more realistic scenarios. We will consider this as our future work.

The use of more complex deep neural networks is also a promising direction. In this paper, we mainly follow previous work to consider ResNet and DenseNet. Using deeper and larger networks can improve the classification performance, and we will also consider them in the future. Moreover, although the use of larger networks is promising, it may also lead to a higher risk of overfitting problems. Then it is helpful to use our proposed model selection criteria for early stopping.