Candidate Label Set Pruning: A Data-centric Perspective for Deep Partial-label Learning

Thank you so much for your meticulous review!

Q1: A small typo? I guess the author may have forgotten to divide n in the formulation of $\beta$ in Definition 1, as $\beta$ does not equal 1 but n in the optimal pruning case. Can the author clarify that?

A: Thanks for pointing out this issue. We have corrected the issue in the formulation of $\beta$ to improve the clarity: $\beta=\frac{\sum _{i=1} ^{n}|\widetilde{Y} _{i}|}{\sum _{i=1} ^{n}(|Y _{i}|-1)}$. In the optimal pruning case, $\beta=1$ satisfies our consideration.

Q2: It is vague to describe PLL in the abstract that "Partial-label learning (PLL) allows each training example to be equipped with a set of candidate labels." Does the author intentionally ignore the assumption that "only one is the ground-truth label"? It is better to clarify as some PLL research wave the limitation to investigate a new PLL task (called Unreliable or Noisy PLL).

A: Thanks for pointing out this issue. We would like to explain that this assumption was formally introduced in the definition of PLL in Section 3.1 where each candidate label set $Y_{i}$ consists of a true label $y_{i}$ and a false candidate label set $Y_{i}^{\prime}=Y_{i}\backslash \{y_{i}\}$. To further improve the clarity of our paper, we have highlighted this assumption both in the abstract and problem setup in our paper.

Q3: Some state-of-art PLL methods are missing in the reference, such as A Unifying Probabilistic Framework for Partially Labeled Data Learning; Mutual Partial Label Learning with Competitive Label Noise.

A: Thank you for letting us know about the related work! We have introduced them [1, 2] in the related work section of our paper.

References:

[1] A Unifying Probabilistic Framework for Partially Labeled Data Learning. TPAMI. 2022.

[2] Mutual Partial Label Learning with Competitive Label Noise. ICLR. 2023.

Thanks for raising this concern. We have already considered this issue in our proposed method. When a PLL instance only has one candidate label, i.e., $|Y_{i}|=1$, the number of eliminated candidate labels $\gamma_{i}=\lceil \tau(|Y_{i}|-1) \rceil$ in Eq. (2) would be always equal to zero, implying that the proposed method would not prune any candidate label of the instance, which is reasonable because the only candidate label is exactly the true label.

Thank you so much for your insightful comments!

Q1: The negative impact of eliminating the correct label from the candidate label set.

A: Thanks for raising the concern. Indeed, conventional PLL methods could suffer from the negative impact of noisy PLL instances inevitably. Our countermeasure to alleviate this issue is to reduce the number of noisy PLL instances in the pruned PLL data as much as possible. For this purpose, we aim to decrease the upper bound of the pruning error by leveraging high-quality representations extracted from advanced feature extractors and using appropriate parameters in the proposed method. Empirically, the pruned PLL datasets only have a very small proportion of noisy PLL instances (as shown in Table 3), which makes PLL methods achieve impressive performance improvements. This observation definitely validates the negligible effect of noisy PLL instances caused by the proposed method. Besides, it would be also interesting to deal with potential noisy PLL instances directly during the pruning procedure.

Q2: More details about the $k$-NN algorithm and feature extractors.

A: Thanks for pointing out this issue. We have provided more details about the $k$-NN algorithm and the used feature extractors in Appendix C of our paper. Specifically, we employed the visual encoder of BLIP-2 [1] to extract 768-dimensional high-quality representations for all training instances in each PLL dataset. Then, we leveraged FAISS [2] to conduct the fast $k$-NN searching for each PLL instance based on the squared Euclidean Distance.

Q3: Consider the work [3] in related works and experiments.

A: Thanks for your suggestion! We have introduced the work [3] into the related work section of our paper. Moreover, we have conducted additional experiments by using the POP method [3] on the original and pruned PLL datasets following its suggested configurations. The experimental results on CIFAR-10 and CIFAR-100 are shown in the following tables. Note that O (vs. P) Test Acc means the test accuracy obtained by training on the original (vs. pruned) PLL dataset.

From the above tables, we can see that POP equipped with our proposed CLSP method has significant performance improvements.

Q4: What is the main difference between a data-centric method and a pre-processing method?

A: Thanks for the interesting question. We think that our proposed data-centric method can be regarded as a special type of pre-processing method. This is because the proposed method can be used to pre-process training PLL instances before using these data to train deep PLL methods. We expect that our first data-centric method will motivate more studies on pre-processing methods for PLL.

Q5: Does the feature extractor in the $k$-NN algorithm come from the classifier during the training process? If yes, it seems unreasonable to say that the method is training-free. If not, a pre-trained model should be introduced.

A: Thanks for raising this concern. We would like to explain that the feature extractor does not come from the classifier during the training process. Instead, the feature extract comes from a multi-modal pre-trained model (e.g., CLIP, ALBEF, BLIP-2), which has not been trained on the corresponding PLL dataset. Hence, the proposed method is indeed training-free by leveraging a multi-modal pre-trained model. We have already introduced the pre-trained models in Section 4.1 of our original paper. Notably, to evaluate the effect of different types of feature extractors, we additionally trained two ResNet-18-based models (i.e., ResNet-SSL, ResNet-S) on the corresponding PLL dataset. Note that we did not use them in the proposed method and just used them for comparing the effects of different types of feature extractors. Specifically, ResNet-S was trained on each PLL dataset with the original clean supervision, while ResNet-SSL was trained on each PLL dataset by the self-supervised learning method SimCLR [4] without any supervision. More details about the training setup of these two feature extractors have been shown in Appendix C of our paper.

References:

[1] https://github.com/salesforce/LAVIS

[2] https://github.com/facebookresearch/faiss

[3] Progressive Purification for Instance-dependent Partial Label Learning. ICML. 2023.

[4] https://github.com/google-research/simclr

Thank you so much for your insightful comments!

Q1: The numerical simulation experiment about the calculating of values and conclusions is not shown clearly enough. (How are these values of $k$ and $\gamma_{i}$ used to calculate the upper bound in Figure 1?)

A: Thanks for pointing out this issue. We have improved the clarity of this part in Section 3.3 of our paper. Our objective of the numerical simulation experiment is to evaluate the effects of representations of different qualities and different label ambiguities on the upper bound of the pruning error, for the proposed method. For this purpose, we empirically set various values of $\delta_{k}$ and $\rho_{k}$ where a small value of $\delta_{k}$ ($\rho_{k}$) implies high-quality representations (candidate label sets of low label ambiguity). By substituting the specific values of $\delta_{k}$ and $\rho_{k}$, we can calculate an exact upper bound with varying values of $k$ and different numbers of eliminated candidate labels $\gamma_{i}$. In this way, we are able to know how to select suitable values of $k$ and $\gamma_{i}$, with different feature extractors and PLL settings in the experiments.

Q2: The PASCAL VOC dataset used in the experiment is not introduced well, as the dataset is not originally for partial label learning. (How is it used in partial-label learning?)

A: Thanks for raising the concern on the used PASCAL VOC dataset. In particular, we followed the previous work [1] to construct the dataset where objects in images were cropped as instances and all objects appearing in the same original image were regarded as the labels of a candidate set. In this way, we could obtain a PLL version of the PASCAL VOC dataset and perform any PLL method on this dataset. More characteristics of PASCAL VOC have been shown in Appendix C of our paper.

Q3: The detail of trained feature extractors (ResNet-SSL, ResNet-S) is shown unclearly.

A: Thanks for pointing out this issue. Specifically, ResNet-S was trained on each PLL dataset with the original clean supervision using the cross-entropy loss, while ResNet-SSL was trained on each PLL dataset by the self-supervised learning method SimCLR [2] without supervision. We have further detailed the training setup of feature extractors in Appendix C of our paper.

Q4: It would be appreciated if a clearer explanation for Definition 2 is provided. Why is this Definition needed?

A: Particularly, Definition 2 aims to characterize the candidate label distribution in the local representation space. Specifically, a PLL instance’s $k$-NN instances are expected to have the true label in their candidate label sets with a high probability, while their candidate label sets are unlikely to have the same false candidate label. This characteristic contributes to the label distinguishability of candidate label sets in the local representation space, which is important for the proposed method to achieve a satisfying pruning error.

Q5: Why does the loss curve on VOC in Figure 5 have a rise (except the PRODEN algorithm)? This phenomenon is different from other cases on CIFAR and Tiny-ImageNet datasets.

A: The phenomenon is true that the training loss curves of certain PLL methods have a rise after training with pruned PLL data on PASCAL VOC in Figure 5. We reckon that there are two reasons. Firstly, there are more proportional noisy PLL instances in PASCAL VOC (the pruning error is 5.2%) than in CIFAR and Tiny-ImageNet (the pruning error is almost < 1%). Secondly, label disambiguation towards PLL instances in PASCAL VOC is more challenging due to the complicated visual objects in PASCAL VOC. Hence, fitting noisy PLL instances in PASCAL VOC is more difficult, thereby leading to a larger training loss value. More details are shown in Appendix D of our paper.

Q6: What is the potential limitation of the proposed approach? Discussing this point is also important to have a comprehensive understanding of the proposed approach.

A: Thanks for raising the concern. We think that a potential limitation of the proposed method is the presence of noisy PLL instances in the pruned PLL dataset whose true label is incorrectly eliminated and inside the non-candidate label set. Besides, many PLL methods are incapable of dealing with noisy PLL instances in the pruned PLL datasets. To alleviate this issue, as analyzed in the numerical simulation experiment, we can decrease the upper bound of the pruning error by leveraging high-quality representations extracted from advanced feature extractors and using appropriate parameters in the proposed method, thereby effectively reducing the number of noisy PLL instances in the pruned PLL dataset. In addition, it would be also interesting to deal with potential noisy PLL instances directly during the pruning procedure.

References:

[1] Long-tailed partial label learning via dynamic rebalancing. ICLR. 2023.

[2] https://github.com/google-research/simclr

Thanks for your insightful comments!

Q1: More empirical analyses in the experiment should be presented.

A: Thanks for your suggestion! We have provided additional analyses on the experimental results of the transductive accuracy comparison in Table 7 and analyses on the training loss curves in Figure 5. For the transductive accuracy comparison in Table 7, we discover that naive PLL methods (e.g., CC, PRODEN, LWS, and CAVL) have more significant performance improvements than advanced PLL methods (e.g., PiCO, CRDPLL, ABLE, and IDGP). We think that this is because naive PLL methods have a limited capability in label disambiguation compared with advanced PLL methods, hence the proposed CLSP method has a bigger effect on promoting label disambiguation in naive PLL methods. For the training loss curves in Figure 5, we found an unusual phenomenon where the training loss values of many PLL methods become larger on PASCAL VOC after training with pruned PLL data. We reckon that there are two reasons. Firstly, there exists a higher proportion of noisy PLL instances in PASCAL VOC (where the pruning error is about 5.2%) than in CIFAR and Tiny-ImageNet (where the pruning error is less than 1% in most cases). Secondly, label disambiguation of training PLL instances in PASCAL VOC is more challenging due to the complicated visual objects in PASCAL VOC. Hence, fitting noisy PLL instances in PASCAL VOC is more difficult, thereby leading to a larger training loss value. More details of these empirical analyses have been shown in Appendix D of our revised paper.

Q2: More explanations on bad cases are needed to show the limitations of the proposed method. (How to explain the phenomenon of bad cases?)

A: Thanks for raising the concern about bad cases! We have explained the potential reason for the based cases in the experiment results of our paper. Specifically, there are a few bad cases (4/149 $\approx$2.7%) in the experimental results where the performance of certain PLL methods has slightly degraded after training with pruned PLL data. We argue that this is because the involved PLL methods (i.e., ABLE and SoLar) in the bad cases have a time-consuming training procedure (500 and 1000 epochs respectively), hence they tend to overfit noisy PLL instances eventually, thereby leading to performance degradation.

Q3: How about pruning on noisy PLL instances?

A: Thank you for this interesting point! Regrettably, the proposed CLSP method may be infeasible to handle noisy PLL instances directly. This is because our method only focuses on the candidate label set and thus has no effects on noisy PLL instances whose true label is inside the non-candidate label set. Fortunately, when the number of noisy PLL instances in the original dataset is not large, the proposed CLSP method can be still applied.

Q4: Could you show the values of $\delta_{k}$ and $\rho_{k}$ on the real-world dataset VOC?

A: Thanks for your suggestion. Following the empirical formulation of $\delta_{k}$ and $\rho_{k}$ shown in Section 4.1, we have conducted additional experiments to calculate their empirical values on PASCAL VOC. The calculated values of (1-$\delta_{k}$) and $\rho_{k}$ are shown in the following tables.

From the above tables, we can see that the values of (1-$\delta_{k}$) and $\rho_{k}$ both decrease progressively as the value of $k$ increases. Further compared with the values on CIFAR in Figure 3, the values of (1-$\delta_{k}$) on PASCAL VOC are smaller than on CIFAR, which implies the quality of representations on PASCAL VOC is lower than on CIFAR. On the other hand, the values of $\rho_{k}$ on PASCAL VOC are larger than on CIFAR in most cases, indicating that label ambiguity in PASCAL VOC is more severe than in CIFAR. Hence, the proposed CLSP method is more challenging to employ on PASCAL VOC.

Q5: A unified proportion $\tau$ for each training instance is used in Eq. (2). Are there other ways to adaptively control the number of eliminated candidate labels for each training instance?

A: The answer is positive! It is feasible to eliminate different numbers of candidate labels in training PLL instances in an adaptive manner. We can formulate the problem of adaptively deciding the optimal number of eliminated candidate labels for each training PLL instance as an integer programming problem. In this case, certain optimization techniques might be required to solve the integer programming problem.