Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations

Thanks for the reviewer's efforts reviewing this paper and acknowledge of our work. We address the concerns as follows.

Given the extensive content, the main text of the paper omits certain details, which may not be immediately straightforward to follow for some parts, e.g., section 3.2.

Thanks for pointing this out. Please understand that this is mainly due to the space limit that we have to include the derivation detail in Section 3.2 into Appendix. We will try to make this more clear in our revised paper.

If you find our revise and response helpful, please consider raising the score for better support of our work.

Thanks for the reviewer's efforts reviewing this paper and acknowledge of our work. We address the concerns as follows.

---

The implementation of EM over all possible labelings may increase the computation time.

For all the weak supervision settings studied in this paper, our method introduce negligible computation time. This is because all the weak supervision considered in this paper impose conditional independent latent ground truth variable. A runtime analysis can be found in Table 21 of our Appendix, where the proposed method is usually faster than previous SOTA methods.

More related works need to be discussed. The author considers the ground-truth or Bayes label distribution as the latent variables and leverages variational inference for estimating. I am not should this strategy is novel enough in the field of variational inference. I suggest the author add more related works to highlight the novelty of their technique.

Thanks for pointing out this. As we discussed in our paper (line 68 - 78), EM solution for weak supervision is long desirable in the community and there are earlier attempts and relevant methods.
However, we would like to highlight that our method is the first method fully based on EM, without other model and loss design, which universally handles multiple and mixed types of weak supervision. This distinguishes us from previous works.

We also added the following extra relevant works of VI and EM on weak supervision to the discussion of our paper. Per discussion above, these method do not rely on EM itself to achieve reasonable performance.

[1] Xu, N., Qiao, C., Geng, X., & Zhang, M. L. (2021). Instance-dependent partial label learning. Advances in Neural Information Processing Systems.

[2] Garg, A., Nguyen, C., Felix, R., Do, T.T. and Carneiro, G., 2023. Instance-dependent noisy label learning via graphical modelling. In Proceedings of the IEEE/CVF winter conference on applications of computer vision.

[3] Elhassouny, A. and Idbrahim, S., 2022, December. Deep learning with noisy labels: Learning True Labels as Discrete Latent Variable. In 2022 14th International Conference on Computational Intelligence and Communication Networks.

The author should give an explanation of why utilizing an EM framework optimizes the variational lower bound. What is its advantage?

The EM algorithm provides a robust method for handling latent variables by decomposing the maximization of the likelihood function into an E-step and M-step. EM algorithm also has provable property of convergence. Besides, the EM framework is inherently designed to optimize the variational lower bound, as we shown in Appendix C.1.

There is also possible improvements over EM, such as ECM [1] and PX-EM [2], which could be incorporated in our method for further improvements. Due to the scope of this study, this is left as our future work.

[1] X. L. Meng and D. B. Rubin. Maximum likelihood via the ECM algorithm: a general framework. Biometrika, 1993.

[2] C. Liu, D. B. Rubin, and Y. N. Wu. Parameter Expansion to Accelerate EM: The PX-EM Algorithm. Biometrika, 1998

Can the framework be extended to handle other forms of weak supervision (such as imbalanced noisy label learning) and how?

Yes. The proposed unified framework can be easily extend to other weak supervision settings. In this study, we mainly consider three types of weak supervision, where the imposed ground truth label is conditional independent given the weak supervision. Our framework can be extended to similar settings without any change, such as positive unlabeled learning,

As for imbalanced noisy label learning, it is a configuration regarding changes in $P(X)$ itself rather than weak supervision $P(I, Y)$. While our method is not specifically designed for such settings (change in $P(X)$), we showed in Table 3 that our method achieves comparable results on imbalanced noisy label datasets (Clothing1M). For more imbalanced noisy settings, we provide an additional comparison here, following the experiment setting of [1], where we introduce 5:1 head-tail class imbalanced into noisy CIFAR-10 and CIFAR-100 with a ratio of 0.2 below.

From the results, we can summarize that previous methods specifically designed for noisy label learning (DivideMix and ELR+) usually present large performance degeneration on imbalanced settings, whereas our method present comparable performance with method specifically designed for this setting (ULC), demonstrating the extensibility of our method. Our method can also be combined with other design or techniques for handling imbalanced data for further improvements.

[1] Huang, Y., Bai, B., Zhao, S., Bai, K. and Wang, F., 2022, June. Uncertainty-aware learning against label noise on imbalanced datasets. In Proceedings of the AAAI Conference on Artificial Intelligence..

Computational efficiency of EM regarding the weakness above. It would be better that the author provide (theoretical or empirical) analyses regarding the computation of the algorithm.

We have empirical analysis about computation efficiency included in our Appendix. The results demonstrate that our proposed method present better efficiency, compared to previous SOTA methods.

---

If you find our revise and response helpful, please consider raising the score to better support of our work. Also please let us know if there are further questions.

Thanks for the reviewer's time reviewing this paper and suggestions on the relevant works. We address the concerns raised as follows.

---

The article's innovation is limited...which suggests that the presented techniques may not be as novel as claimed.

Thanks for mentioning these relevant works. We need to emphasize that, although there are existing methods that share the similar variational approximation for handling weak supervision (as we also discussed in our paper), there are three fundamental differences and improvement of our method:

• The algorithm/loss level: We provide a new perspective deriving the loss objectives directly from the EM view of various kinds of weak supervision. Most of the existing works use the EM/ELBO to explain their proposed methods, which means their methods are not originated from EM and usually rely other sophisticated designs. Instead, our framework only uses the derived loss functions without other sophisticated design while achieving comparable performance.
• Our general nature: Our framework is the first one that can naturally and universally handle the mixture of various weak supervision settings.

In summary, most of weak supervision methods can be explained/understood from the EM perspective, however, our method is naturally derived from and only rely on the EM, and applicable to various settings. We will include the mentioned works into discussion in our revised paper.

[1] Xu, N., Qiao, C., Geng, X., & Zhang, M. L. (2021). Instance-dependent partial label learning. Advances in Neural Information Processing Systems, 34, 27119-27130.

[2] Yao, Y., Liu, T., Gong, M., Han, B., Niu, G., & Zhang, K. (2021). Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34, 4409-4420.

Some important baselines should be compared, such as [1,2] in SSL.

Thanks for mentioning these relevant baselines. We omitted some baseline comparison on the paper mainly due the space limit. More baselines of all settings can be found from our Appendix. For the mentioned SSL baselines, we provide a brief comparison of error rate below:

Note that this comparison is based on TorchSSL benchmark, instead of USB. We will include these baseline comparison in our revised paper.

[1] Nguyen, Khanh-Binh, and Joon-Sung Yang. "Boosting Semi-Supervised Learning by bridging high and low-confidence predictions." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Schmutz, Hugo, Olivier Humbert, and Pierre-Alexandre Mattei. "Don’t fear the unlabelled: safe semi-supervised learning via debiasing." The Eleventh International Conference on Learning Representations. 2022.

Given that the method of using...beyond these existing approaches in terms of innovation or application?

As discussed earlier, our method distinguish from existing approaches in the following aspects:

• Earlier attempts usually require additional assumptions (line 68 - 71) to handle a single type of weak supervision, whereas ours does not and naturally and universally handles various type of weak supervision.
• Unlike most of previous methods, our proposed method is naturally derived from EM and does not require other sophisticated design (model architecture, extra loss function, etc.). This demonstrates the scalability of our method.
• The proposed method naturally extends to mixed weak supervision settings and also other weak supervision settings not studied in the paper. We need to highlight that this is the first method that handles mixed weak supervision settings robustly using a unified framework.

Could you provide a detailed analysis explaining why the unified framework demonstrates superiority over specifically designed approaches?

Thank you for this insightful question. Our unified framework demonstrates superiority over specifically designed approaches due to several key factors:

• Simplicity and generalizability: Unlike many existing methods that require complex and specifically tailored architectures or additional loss functions for each type of weak supervision, our approach leverages a unified EM-based framework that is naturally derived without needing additional sophisticated design elements. This simplicity allows for a more straightforward implementation while still achieving competitive or superior performance across different types of weak supervision settings.
• Versatility in handling mixed supervision: Many existing approaches are tailored to handle only a single type of weak supervision, often requiring additional assumptions or modifications to the model when dealing with different forms of supervision. In contrast, our framework is the first to naturally and universally handle mixed types of weak supervision without requiring any additional assumptions or modifications. This versatility makes our method more robust and adaptable to various real-world scenarios where mixed supervision is common.
• Scalability and robustness: The unified nature of our framework ensures that it can be easily extended to new and unseen weak supervision settings without needing to redesign or retrain the model from scratch. This scalability is a significant advantage over specifically designed approaches, which often struggle to maintain performance when extended to different or more complex weak supervision scenarios.
• Theoretical foundation: Our method is firmly grounded in the Expectation-Maximization (EM) algorithm, providing a solid theoretical foundation that ensures the method's validity and robustness.

---

If you find our revise and response helpful, please consider raising the score to better support of our work.

We thank the reviewer for the detailed and informative review. We recognize several of the concerns expressed by the reviewer and agree that addressing these will indeed enhance the paper. We have attempted to do so below. We hope this is satisfactory.

---

Neither abstract...

The setting studied in this paper is multi-class classification, as is common for almost all reported work on weak supervision, especially baselines covered in this paper.

This stands in contrast..it restores identifiability of the model.

Please note that the model is not dependent on the assumption of a unique label. In the general case we are given a label vector $Y \in \mathbb{R}^C$. For unique labels, $Y$ is one-hot. For uncertain labels, it will be dense. The loss computation remains the same in both cases, and the EM solution still holds.

• Specifically, for noisy labels, label noise can be captured by a noise-transition matrix $\mathcal{T}(\hat{Y}|Y;\omega)$ (Eq 8). The noisy label $\hat{Y}$ is simply $\hat{Y} = \mathcal{T} Y$. The EM solution works with $\hat{Y}$ to infer $\mathcal{T}$ and the model parameters under $Y$. The framework holds regardless of whether $Y$ is one-hot or dense.
• Even with one-hot $Y$ the model is non-identifiable. Consider any permutation matrix $P$. We have $\hat{Y} = \mathcal{T}Y = (\mathcal{T}P)(P^T Y)$, i.e. it is not permutation invariant. Consequently, the model is not identifiable. This is regardless of whether $Y$ is one hot or dense.

A serious...vague:

As explained above, we model $P(X, I; \theta)$ in Eq4. The missing data here are $Y$ as evident from Eq 4 and 5. This level of description fully specifies the EM problem, although for the specific update rules the form of parameterization becomes important to state. In our case, we use the decomposition of Eq 5, wherein we model $P(Y|X; \theta)$ using neural nets, leading to the solutions of Eq6-10.

For partial labels...

The EM solution generalizes to any $P(Y|I)$.

Specifically, we study two types of partial labels: instance-independent and instance-dependent ones. The label candidates of the first follows the uniform assumption; the second follows the instance-level strategy. On both types of partial labels, our method presents superior performance (Tab 1 and 9).

unlabeld data…

For unlabeled data, $P(Y|I;\theta)$ is also learned (Eq 7) as in the case of the partial label. Our methods perform robustly with different numbers of unlabeled data (Tab 2). Note that our solution does not preclude the use of a known or assumed prior $P(Y|I)$ (e.g. uniform), in which case this distribution would not be learned.

noisy labels…

For noisy labels, $P(Y|I,\theta)$ is modeled through a noise transition matrix, which is learned, as explained in (Eq 8 and 9).

l. 170: Note that P (X; θ) is omitted...assumption?

The primary component of the model learns the mapping $P(Y|X)$, since this is not a generative model for $X$. As such, $P(X)$ need not be modeled. This assumption is common in the training of discriminative models.

$P(Y|X)$ in turn may be decomposed into $P(\{Y_i\} | X)$, which permits modeling of the distribution of subsets of components of $Y$. The ``may’’ merely includes this possibility.

This is not correct...simplified.

As stated in the paper, whether $P(I|Y, X;\theta)$ reduces to $P(I|Y)$ depends on the nature of I. If I is directly drivable from Y, i.e, $I=f(Y)$, then $P(I|Y,X;\theta) = P(I|Y)$ e.g. $I = Y$, $I = count(Y)$, etc. We have also considered other settings where $I$ is dependent on $X$, and this is not reducible, as in Tab 3, 9, and 15. E.g. in noisy labels, it is treated as a noise transition model.

Equation (6) seems to...equations.

$\mathcal{A}_s$ and $\mathcal{A}_w$ represent strong and weak augmentation, and are actually defined early in the paper before Eq 1 (L101-102). Besides, from C.2, from Eq13 to 14, what we did is replace the random variable with actual data notation to make it a true loss function. If the reviewer thinks this is still vague, we will try to make it more clear in our revised paper.

l. 174: For independent instances...up

The typical setting is where the data may be viewed as a collection of independent instances. In other settings (e.g, streaming), the labels in some instances may bear statistical dependence on those in other instances. Our framework permits both settings and experiments including both instance-independent and dependent settings. We will try to make this more clear in the revised paper.

l. 223 Things become more...paragraphs.

In the noise-free case, $I$ presentes uncorrupted information about $Y$. E.g., it might state that a collection of instances includes a particular label, without identifying the specific instances that do. While the label information provided is incomplete, what is provided is true.

In the noisy case, an instance labeled as $y$ does not imply that its label is actually $y$. We will clarify this in Sec 4.3.

Equations (6), (7), (8) contain data...method.

As mentioned in the paper, these are common techniques in this field. Besides, we have an ablation in Tab 19 of Appendix on strong-weak augmentation. Removing strong-weak augmentation leads the baselines to significant performance drop, whereas the effects on ours are much smaller.

This argument doesn't … O(1) operation.

Thanks to the reviewer for pointing out this error. We were trying to convey that for closed-form solutions, once sufficient statistics are gathered, the subsequent estimator has a fixed complexity, whereas for iterative solutions this is not true. However, the actual computation for gathering statistics would be $\mathcal{O}(N)$ as in other models. We will make this more clear in our revised paper.

We hope our above response addresses the reviewer’s concerns. If you have further concerns, please do not hesitate to let us know:)

Here we address other concerns from the reviewer due to limited space in rebuttal.

The footnote...P(Y|X;\theta)?

Thanks for pointing this out. In the noisy label setting, this is not the case, where the noise transition model also depends on $\theta$ and $X$, as we discussed above.

Typos/grammar: l. 87...

Thanks for pointing out these typos. We have fixed them in our revised paper.

L68 - 71 seem...scalability.

The "scalability" in the limitation section is meant for larger datasets such as ImageNet-1K that we have not investigated due to the heavy computation cost. However, our method indeed demonstrates scalability on various settings where the runtime advantage compared to previous SOTA methods is shown in Table 21.

Dear Reviewer Z7AB,

As the discussion period coming towards the end, we would appreciate if you review our response and we hope it will help resolve your concerns. Also please do not hesitate to let us know if there is any more question. Your insights have been invaluable, and we thank you sincerely for your time and effort.

Best, Authors.

Identifyability: If the model is non-identifiable, in particular, if you could put an arbitray permutation matrix on the labels, then no model could hope to learn something useful, no? So you generally need to put some assumptions/inducitve priors on the noise transition matrix; e.g., if labels are deterministic, then I think having the largest entry in each row if $\\mathcal{T}$ be the diagonal restores identifiability, but that condition wouldn't be sufficient with randomized labels.

---

The property of the second term $\\log P(I|X, Y ; θ)$ is dependent on the nature of imprecise label I. If I contains information about the true labels Y, such as the actual labels or the label candidates, it can be reduced to P (I|Y ), i.e., the probability of I is no longer dependent on X or θ and thus can be ignored from Eq. (5).

To clairfy, what I was objecting to here was the writing of the paper, not the statement that $P(I|Y,X;\theta)$ can be reduced to $P(I|Y)$ in some settings: The paper writes If I contains information about the true labels Y, which is a much weaker condition than what you stated in your rebuttal, if $I$ is directly drivable from $Y$. (this phrase is even repeated later in the paper, Since S contains the information about the true label Y...)

Note that this step also implicitly assumes that $Y$ is a one-hot vector; otherwise, if it has non-zero weight for labels from two different bags, $I$ is no longer a measurable function of $Y$.

---

I don't think eq. (4) models $P(X, I;\theta)$; this equation is just the maximum-likelyhood principle and the law of total probability; it doesn't actually tell me anything new about $P(X, I;\theta)$ In fact, I think this goes together with my later point. One of the modeling assumptions is $P(X, I;\\theta) = P(I | X; \\theta) P(X)$, i.e., the parameters $\\theta$ are not used to model the distribution over $X$.

---

You are right, $\mathcal{A}_s$ and $\mathcal{A}_w$ are mentioned earlier in the paper; however, I don't think they are actually defined; they appear in a sentence discussing the Pico method

It optimizes the cross-entropy (CE) loss between the prediction of the augmented training sample Aw(x) and the disambiguated labels s. PiCO learns a set of class prototypes from the features associated with the same pseudo-targets. A contrastive loss, based on MOCO [67 ], is employed to better learn the feature space, drawing the projected and normalized features zw and zs of the two augmented versions of data Aw(x) and As(x) closer.

This does not actually give any information about what $\mathcal{A}_w$ and $\mathcal{A}_s$ are. More importantly, though, the way I read the paper, this is a description of a trick used to improve performance.

But then in 3.1, you try to derive a general framework from first principles, which then is applied in 3.2 to these various settings. This is what happens in C.2. In the main paper, however, it is not equation (14) that is presented, but (6) which contains these additional augmentations. There is no mention of these in the preceding section 3 or the derivation in C.2; no reason given why two different agumentations should be used in this cross-entropy formulation. I'm still not quite sure what $p(y|A_w(x), s; θ_t)$ actually is. If I have four classes, and $s\_1=\\{1, 2\\}, s\_2=\\{3, 4\\}$, and $s=1$, according to the text, $p$ would be something like $\\{0, 0, 0.5, 0.5\\}$; i.e., zero on the classes that are consistent with $s$ and uniform (in the instance-independent case) on the others???

---

In the noise-free case, presentes uncorrupted information about. E.g., it might state that a collection of instances includes a particular label, without identifying the specific instances that do. While the label information provided is incomplete, what is provided is true. In the noisy case, an instance labeled as does not imply that its label is actually . We will clarify this in Sec 4.3.

I still don't understand this. In the noise-free, partial label case, the observation tells you some distribution over possible labels. In the noisy-label case, the observation also gives you a distribution over labels. In both cases, you do get some probabilistic information about the true label (assuming an underlying deterministic labeling).

---

I don't really see the novelty in the described approach. If I come upon an incomplete labelling scenario for the first time, what would be the advantage of me reading this paper, compared to just reading about EM in general? Maybe I'm just missing something fundamental that the other reviewers seemed to get (note my low confidence score).