Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations

Thanks for your valuable feedback. In fact, we found that most of the weakness mentioned here has already been discussed in the Appendix with detailed results. Here, we provide a more detailed explanation and point out the related sections in our Appendix.

1. "The article's innovation is limited, as the approach of considering ground-truth labels or Bayes label distribution as latent variables and using variational inference for approximation in weakly supervised learning is already a common method[1-2], which suggests that the presented techniques may not be as novel as claimed."

Thank you for mentioning these related works, and we have included them in our related work. Indeed, our contribution is certainly not to present the first interpretation of variational inference on the latent ground-truth labels, but to present the first complete EM view without any approximation/assumption on the data, thus it can handle different types of imprecise labels and even a mixture of them robustly. For example, in [1], extra prior distribution is needed in the variational approximation and applies to only partial labels. In [2], it also requires an encoder-decoder similar architecture to capture the distribution of the latent variable for noisy labels.

Beyond the novelty in unifying existing settings on PLL, SSL, and NLL, our framework can be easily implemented in different settings and achieve SOTA results with faster convergence. We hope that our contributions can be understood and valued in this way.

[1] Xu, N., Qiao, C., Geng, X., & Zhang, M. L. (2021). Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34, 27119-27130.

[2] Yao, Y., Liu, T., Gong, M., Han, B., Niu, G., & Zhang, K. (2021). Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34, 4409-4420.

2. "The article's treatment in section 3.2, where "P(S|X, Y ; θ) reduces to P(S|Y ),...instance-dependent partial label learning."

Thanks for mentioning the instance-dependence situation. In fact, we think there is a little misunderstanding here. As mentioned in the last paragraph of Section 3.1, $P(I|X, Y;\theta)$ depends on the nature of imprecise information $I$. If $I$ is not instance-dependent, it can be largely ignored. If $I$ is instance-dependent, it should be maintained and optimized as a supervised term. In fact, in Section 3.2, $P(I|X, Y;\theta)$ is indeed maintained for noisy label learning and semi-supervised learning. Its performance is verified on instance-dependent noisy label learning as shown in Table 3 and Table 15. For instance-dependent partial label learning as in [1, 2], it should be also maintained as $P(S|X, Y ; \theta)$. Here we provide a comparison of our method to [1, 2] in the benchmark of instance-dependent partial label learning. We follow the training recipe in [2] to train our methods and report the average accuracy of 3 runs.

The discussions and results on instance-dependent partial label learning are included in the revised Appendix D. 3.2.

[1] Xu, N., Qiao, C., Geng, X., & Zhang, M. L. (2021). Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34, 27119-27130.

[2] Xu, N., Liu, B., Lv, J., Qiao, C., & Geng, X. (2023). Progressive purification for instance-dependent partial label learning. In International Conference on Machine Learning (pp. 38551-38565). PMLR.

[3] Wu, D. et al. (2022). Revisiting Consistency Regularization for Deep Partial Label Learning, ICML.

We deeply appreciate the increased rating. In terms of the EM formulation, we totally agree that there exist previous methods treating the true labels as latent variables and even utilizing variational inference as ours. However, note that for our EM formulation, the significant difference and innovation lie in that we do not require any assumption on the probability, e.g. conditional independence between Y and I or X (with our NFA modeling), whereas previous methods usually do. This flexibility allows our approach to generalize to many different settings and different types of data. We hope we have made this point more clear. Thanks again for your efforts in improving this work.

Thanks for your recognition of the significance of our work and your valuable suggestions. Our response is as follows,

1. "It is good to see the proposed method can be seamlessly combined with the data augmentation techniques, but it would be helpful to examine the model's performance without data augmentation techniques. It seems that the method's performance is sensitive to the quality of data augmentation, but not all kinds of tasks can be easily benefit from data augmentation. An ablation study would be helpful."

Thanks for this valuable suggestion. We totally understand your point that strong augmentation might be difficult to apply such as on text data or audio data. However, strong augmentation, proposed in FixMatch [1], has been demonstrated very important and critical for achieving effective performance in SSL. For PLL, many baseline methods also use strong augmentations. For NLL, the strong augmentation is less important compared to other settings. Thus, we provide an ablation study of the strong augmentation for PLL and NLL:

For PLL, it is shown that the strong augmentation indeed affects the performance very much for both the baseline PiCO and our method. For NLL, removing the strong augmentation has less effect on our method but has a detrimental effect on SOP. We have included the above ablation study in Appendix D. 7.

2. "There are also some other related works on unifying multiple problem settings of weakly/imprecise supervised learning. Some discussions on this topic can improve this paper. For example, [1] Centroid Estimation With Guaranteed Efficiency: A General Framework for Weakly Supervised Learning, TPAMI [2] Weakly Supervised AUC Optimization: A Unified Partial AUC Approach, arxiv 2305.14258"

Thanks for mentioning these related works and we have added discussion about them in the revised version of related work.

3. "The presentation can be further improved. There are typos and errors in the paper, e.g., missing punctuations, broken cross references to the figures, etc."

Thanks for pointing this out. We are sorry for the errors and typos in the paper and the confusion caused. We have updated the errors and typos we found and fixed the broken references in the revised paper.

We hope our response can resolve your concerns.

1. "The experimental data in the current article are obtained from balanced and relatively small datasets, lacking sufficient experimental evaluation. At the same time, the article does not discuss the computational complexity or scalability of this framework. Although the article mentions the limitations of some previous methods in dealing with specific forms of imprecise labels, it does not provide a detailed discussion on the scalability of this framework in large-scale datasets or complex scenarios."

Thank you for your valuable feedback.

We compare our method with previous baselines on the standard benchmarks of each setting, which are commonly used in prior works. In NLL, Clothing1M and Webvision are indeed large-scale datasets with realistic instance-dependent noise. Additionally, to resolve your concern in large-scale datasets, we also provide the ImageNet results of our method for semi-supervised learning which we did not include in our main paper before:

As for imbalanced and open-set settings which are beyond the scope of our research purpose to unify SSL, PLL, and NLL, we would like to point out that applying our framework to these settings is as challenging as applying existing previous ones because learning on various restricted $X$ is a different problem of learning on various imprecise $Y$. The study of transferring to restricted data distribution requires an additional large amount of work and is out of the scope of our current study, where our main focus is the unification of learning with different imprecise labels, instead of different data. These two settings have been known to be challenging in the ML community for years, and they could perhaps not be solved in one framework. We intend to look after them in the future work.

Regarding the computation complexity, since our method is quite simple and the loss function is derived as closed-form, our algorithm is usually faster than the baselines. We provide a run time analysis to verify it here:

Note that the runtime is averaged over all training iterations; thus, the performance gap is significant. We have included the runtime analysis in the revised Appendix D.8.

2. "Moreover, in terms of the presentation of the paper, the author's description of existing methods is not clear enough. The paper uses a large number of formulas and tables for description, lacking visual explanations. The comparison of the experimental results also appears vague and unclear."

We appreciate your feedback on the presentation of our paper. We recognize the importance of clear description, especially for the existing complicated methods. Please understand that this is mainly because we intend to provide a comprehensive evaluation across many different settings to show our method could cover various imprecise labels. To improve the clearness of the experimental results, we have included an overview visual comparison with a bar plot to demonstrate the significance of our method in revised Appendix D. 2. The results are computed as the average accuracy of different settings for each dataset. It is demonstrated that the proposed method, in general, achieves better performance compared to recent SOTA baselines or achieves performance comparable to them.

Thank you again for your insightful suggestions regarding the improvement and expansion of our work.

In our previous response, we hope we address your concerns regarding large-scale experiments and the analysis of time complexity. As for the discussion on the potential for local optima in the EM algorithm, we did not encounter any convergence issues across the various settings we examined in our study. However, we acknowledge the importance of this aspect and agree that incorporating regularization and conducting a theoretical analysis on this topic would be valuable for future work. Furthermore, exploring the extension of our methodology to imbalanced or other specific data distributions presents intriguing directions for future research.

We are open to further discussions and would be more than willing to address any additional questions or clarifications you might have. Your continued feedback is greatly appreciated, and we look forward to any further insights from you.

4. "In Appendix D, all modifications to the training are mentioned without explanation and motivation."

We are sorry for this confusion. Again, please understand that this is mainly due to the space limit. We would like to provide more explanation and discussion here.

In fact, the two techniques we mentioned in Appendix D.1 are common techniques in each setting. For example, in SSL, strong-weak augmentation is an important strategy for SSL algorithms widely used in existing works such as FixMatch and FlexMatch [1,2]. Thus, it is important to adopt strong-weak augmentation to achieve better performance in SSL. This is similar in PLL settings (PiCO [3] also used strong augmentation). Strong-weak augmentation and entropy loss are also adopted in SOP [4] of NLL. However, we found these techniques are less important for our formulation of NLL. We provide a brief ablation study on the entropy loss of SSL, and both techniques for NLL and PLL here to demonstrate our discussions above.

From the ablation on entropy loss of SSL, it shows that entropy loss only affects the performance trivially.

From the ablation on entropy loss and strong augmentation of PLL, we can observe that strong augmentation is important for both PiCO and our method to achieve better performance. Entropy loss has minimal effect on our method.

From the ablation on NLL, it is shown that the performance SOP is subjected more to the entropy loss especially for strong augmentation. Removing these techniques has a less detrimental effect on the performance of our method.

We have included the above ablation study in Appendix D. 7.

[1] Kihyuk Sohn et al. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

[2] Yidong Wang et al. FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling.

[3] Haobo Wang et al. PiCO: Contrastive Label Disambiguation for Robust Partial Label Learning.

[4] Liu Sheng et al. Robust Training under Label Noise by Sparse Over-parameterization.

5. "NIT: Broken reference"

We apologize for the oversight regarding the broken reference and appreciate your attention to this detail. We have carefully reviewed and corrected all references in the revised version of the paper to ensure they are complete and accurate.

6. "In some experiments, the fully supervised model...why it achieves better performance?"

We appreciate your observation regarding the performance of the ILL framework compared to fully supervised models in Table 1 of PLL. Our hypothesis is that the EM consistency term of ILL may lead to better generalization and robustness. Similar observations can be also found in SSL works such as in SoftMatch [1].

[1] Hao Chen, et al. SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning.

If you think our response resolved your concerns and questions, please consider increasing the rating.

We appreciate your valuable feedback and we are sorry for the confusion caused for you. We hope our response in the following could make our method more clear.

1. "The work ... it seems to me more like the outline than a concrete solution that can be implemented in different ways (what authors mention In the paper)."

Sorry for the confusion caused. Our work is certainly not just an outline, but with novel design and concrete solutions to achieve promising results. We intend to demonstrate the universality and generality of the proposed method so we have to include many different settings into the paper, which might overlook some details of our proposed framework. Here, we highlight our contributions more clearly:

• At a conceptual level, we proposed a unified EM framework that accommodates any imprecise labels by taking the imprecise information $I$ abstractly. We demonstrated its effectiveness on various settings, including PLL, SSL, NLL, and the mixture of them.
• At a solution level, thanks to the unified formulation (Eq. 5), our derived formulations and algorithms in each setting are clearly different from previous baselines. Indeed, it can be implemented in different ways when handling different types of $I$, but in general they all share the same EM formulation by replacing the $I$ with actual imprecise labels.
• Compared with existing baselines: previous baselines in each individual setting can be treated as (usually) simplified versions of our framework. This is probably why our method looks like an outline of previous methods while it is certainly not. On different benchmarks of each setting, our unified framework presents promising results. More importantly, without changing the framework, our method can handle more complicated scenarios robustly, where previous specifically designed methods fall short in performance, as shown in Table 4.
• Extend to broader scenarios: our framework can naturally extend to more imprecise configurations except for the settings discussed in our main paper. This is mainly achieved by the NFA modeling of imprecise label $I$, where NFA can represent each form of imprecise label setting and the proposed EM framework can be conducted on the trellis of NFA with linear time complexity. We omit the NFA details in main paper because the studied settings have rather simple NFA and thus their formulations can be derived in closed-form.

2. "Unfortunately, the main paper is very sparse in details about the actual implementation of many of its elements."

Thank you for your feedback. We are sorry if we missed any details in the main paper as we have to move a significant amount of details to the appendix due to space limit. Please understand that our intention was to present the framework in a manner that highlights its adaptability and generalizability across different settings.

In the revised version of the paper, we have included more concrete and detailed derivations of how the framework can be applied in various scenarios in Appendix C, thereby providing a more comprehensive understanding of its practical implementation. The source is also included in our submission for reproducibility. Hope the above revision can provide more details of the proposed framework and its application to each setting.

3. "More details can be found in Appendix C and D. It is unclear to me when NFA from Appendix C.3 is used in the main paper or not."

Thanks for pointing out the NFA. Our idea of NFA is that any imprecise label information $I$ can be represented as NFA, and the trellis expanded from the NFA can be used to compute the expectation over $P(Y|I, X;\theta^t)$ in EM formulation, as shown in Appendix C.1 and C.2. The NFA representation of $I$ in general can be reflected as all possible labelings imposed by $I$, where we can conduct the EM on it. The benefit of designing the NFA representation is to demonstrate the adaptability of the proposed method because we can represent any type of imprecise label as an NFA and thus accommodate into our proposed framework.

In the settings studied in this paper, the NFA representation of PLL would be the candidate labels as transition paths and each sample as states. The NFA of NLL and SSL would just be all labels as transition paths and each sample as states, since the imprecise label information $I$ here does not constrain any labeling. Since the settings studied in this paper have relatively simple NFA representations, we can directly derive the closed-form formulation of $P(Y|I, X;\theta^t)$ from Equation 5 as we did in the main paper. It can be easily extended to other settings such as multiple instance learning, learning with count/proportion, etc, by constructing a more complex NFA of each case, and actually using a forward-backward algorithm on it to compute EM.

We have explicitly included the above discussion in the NFA section in Appendix C. 6 and hope it can make the NFA interpretation more clear.