Learning the Latent Causal Structure for Modeling Label Noise

Response to Reviewer 22VP

Q1: The structural causal model of the labeling process is not reasonable. The example feature $X$ should be the cause of the noisy label $\tilde{Y}$. If only the clean label y is the cause of the noisy label, then this structural causal model is still modeling class-dependent noise rather than instance-dependent noise.

A1: We believe that it is a misunderstanding. Our model indeed addresses instance-dependent noise. Specifically, the generation of instance $X$ and noisy label $\tilde{Y}$ involves common latent causes $\mathbf{Z}$, such as $Z_3$ depicted in Figure 2(b). In causality theory, the presence of common causes between two variables, such as $X$ and $\tilde{Y}$, establishes dependence between them [1,2]. This indicates that our model is inherently instance-dependent, not instance-independent.

Moreover, when there exists label noise, the real cause of the noise is some latent causal factors rather than the image itself, which is very intuitive. For example, we usually say that light and angle cause noise, but we do not say that the image causes noise. This implies that the direct causes of label noise are light and angle rather than image. However, previous work [6] ignores this point. In their generation model, instance $X$ is the direct cause of the noisy label $\tilde{Y}$. This is one of the insights of our paper. We have also provided experiments to verify the effectiveness of our structural causal model, as shown in Table 3 and 4 of the PDF. The experiment details are in the response to Reviewer S6tG.

Q2: In line 150, the assumption that latent noise variables are generated by the label is not reasonable.

A2: Thank you for highlighting this concern. It is a challenge to identify the latent factors where these factors are not independent but causally related. The assumption that latent noise variables are generated by the label can provide a theoretical guarantee of identifiability. Though this assumption introduces certain constraints, compared to other models like CausalNL, our generative framework offers a more realistic representation of the underlying data generation processes.

To empirically validate our model, we conducted experiments by integrating our generative framework into CausalNL and InstanceGM. These experiments were carried out on the CIFAR-10 dataset, which features instance-dependent label noise. We refer to these adaptations as "CausalNL'" and "InstanceGM'", respectively. The experiment results are shown in the Table 3 and 4 in the PDF. The experiment results demonstrate the effectiveness of our model.

Q3: The CSGN method, like other noise-labeled learning methods based on semi-supervised learning, relies on the small-loss criterion and clean samples. Can the authors show the model performance without using semi-supervised learning?

A3: Thanks for your advice. The CSGN can also work with the early stopping method, PES [5]. We follow the setting in PES and conduct experiments on the CIFAR10 dataset. We refer to the version of CSGN that works with PES as “CSGN-PES”. Notably, CSGN-PES does not rely on semi-supervised learning, the small-loss criterion, or the selection of clean samples. The experiment results are shown in Table 11 of the PDF. The experiment results demonstrate that CSGN-PES maintains robust performance even in the absence of semi-supervised learning techniques.

Q4: The CSGN method introduces semi-supervised learning techniques for warmup; however, the experimental section does not include ablation studies.

A4: Thank you for pointing this out. We have conducted ablation studies to assess the impact of removing the semi-supervised learning warmup phase from the CSGN method. In these studies, we replaced the semi-supervised warmup with a regular early-stopping approach, where the neural networks were trained for 10 epochs on the training data. The variant of CSGN without the semi-supervised learning warmup is denoted as CSGN-WOSM. The experiment results are shown in Table 9 of the PDF. The results indicate that CSGN retains its effectiveness even without the semi-supervised learning warmup.

Response to Reviewer pfn2

Q1: The transition matrix is different sample by sample in instance-dependent transition matrix modeling. It means there is no similarity assumption for instance-dependent transition matrix.

A1: We believe that there is a misunderstanding. The transition matrix varying from sample to sample does not imply the absence of similarity assumption. For example, part-dependent label noise [3] represents an instance-dependent label noise scenario where it is assumed that transition matrices for certain parts of instances are similar. Additionally, without some form of similarity between samples, it is impossible to estimate transition matrices for the entire dataset, as only the noisy label is observable, and the clean label remains hidden. Therefore, existing methods employ predefined similarities to estimate transition matrices across a dataset.

Q2: Any examples of similarity-based assumption failure cases? Can modeling latent causal structure solve those failures?

A2: Yes, our model performs well when the similarity-based assumption fails. We conducted an experiment on the moon dataset. We synthesize the noisy labels where the transition matrices on the same manifold are not the same. The test accuracy of our method is 98.07±0.69%, and the estimation error of the transition matrix is 0.10±0.07. In contrast, the test accuracy of MEIDTM is 91.06±0.75%, and the estimation error of the transition matrix is 0.45±0.16. The results show that our method surpasses the MEIDTM in terms of test accuracy and estimation errors in the transition matrix.

Furthermore, we tested our model in a scenario where the similarity-based assumption is valid, i.e., the transition matrix is the same across the same manifold. The test accuracy of our method is 98.35 ± 0.19%, and the estimation error of the transition matrix is 0.08±0.05. In contrast, the test accuracy of MEIDTM is 96.23±0.25%, and the estimation error of the transition matrix is 0.42±0.11. The experiment results show the generalizability and robustness of the proposed method.

Q3: The paper assumes that the relation structure is established from the causal structure. It means there exists some arbitrary similarity assumption of similar latent leads to similar transition matrix. The similarity in manifold and the similarity of latent structure could be very similar, since the samples generated from the same latent variable would have similar features.

A3: We do not manually define some similarities like the existing methods. Instead, our method aims to recover the causal factors that cause noisy labels, and the relations among different transition matrices are established based on the causal factors. For example, an annotator is likely to annotate the pictures with the feature “furs” as “dog”. When this annotator annotates the pictures containing the feature “furs,” their noisy labels are probably “dog.”

The similarity of our model is not similar to the similarity in the manifold since our method can still perform well when the similarity assumption in the manifold is not met.

Q4: The motivation to learn the latent causal structure is not convincing enough.

A4: Learning the latent causal structure can recover the causes of noisy labels and then establish the relations among different transition matrices without predefined similarity. The empirical results in A2 also demonstrate the effectiveness of our method.

Q5: The proposed method will require much time and memory computation cost since it should generate X, which is a limitation of this method.

A5: Our method is a generative-based method, and it requires an additional generative network, leading to more computation cost. We acknowledge this limitation and will include it in Appendix C. However, the existing work, CausalNL and InstanceGM, also have this limitation.

Q6: The baselines are outdated. Since this paper utilizes the generative model for classification task, NPC and SOP are important baselines.

A6: We have updated our set of baselines. RENT [4], which was published at ICLR 2024, is included. Additionally, NPC and SOP have been included as baselines as well, as shown in Table 1 and 2 of the PDF.

Q7: The position of Figure 5 is inadequate. It should be located at the experiment part.

A7: We will relocate the position of Figure 5 in the final version.

Q8: No ablation studies.

A8: We have conducted ablation studies, shown in Table 9 and 10 of the PDF.

Q9: How can I know the learned latent causal structure is good? Also, how can I measure that a good latent structure will lead to good transition matrix estimation?

A9: The efficacy of the learned latent causal structure can be empirically validated by comparing the performance outcomes of our model against those of other models. Specifically, we conducted experiments using the moon dataset, where our method achieved a test accuracy of 98.07±0.69% and an estimation error for the transition matrix of 0.10±0.07. In comparison, the CausalNL model, which does not model the latent causal structure, recorded a test accuracy of 97.88±0.75% and a transition matrix estimation error of 0.12±0.06. The results show that learning latent causal structure is good, and it can lead to good transition matrix estimation.

Q10: The transition matrix estimation error of MEIDTM is smaller than the CSGN on the CIFAR100 dataset. But the accuracy of MEIDTM is not better than CSGN.

A10: Thanks for your comment. There are other factors that can affect the test accuracy. For example, MEIDTM only employs one neural network to select clean data and trains the classifier. Moreover, MEIDTM does not employ the semi-supervised learning technique. CausalNL is the closest one to our setting. When the settings of the two methods are similar, the one with a smaller transition matrix estimation error has better accuracy.

Dear Reviewer pfn2,

We sincerely appreciate your valuable time and effort in reviewing this paper. As we approach the end of the author-reviewer discussion period, we would be grateful for any additional feedback or confirmation regarding whether your concerns have been satisfactorily addressed.

Note that our rebuttal carefully addresses your concerns. Some major responses are summarized as follows:

1. we explained that the transition matrix varying from sample to sample does not imply the absence of similarity assumption, and we gave an example. More details can be found in the A1 of the rebuttal;

2. we conducted experiments on the scenarios where the similarity-based assumption fails and the similarity-based assumption is valid. The experiment results show that our method can work well in both scenarios, but the performance of MEIDTM drops significantly in the scenario where the similarity-based assumption fails. More details can be found in the A2 of the rebuttal;

3. we empirically verify that the similarity in manifold and the similarity of latent structure is different. Instead of predefined similarity, our method can capture the relations (or similarity) among different transition matrices. More details can be found in the A3 of the rebuttal;

4. we have added RENT (published at ICLR 2024), NPC, and SOP as baselines, and their empirical results are shown in Table 1 and 2 of the PDF;

5. we have conducted sufficient ablation studies, as shown in Table 9 and 10 of the PDF;

6. we have empirically verified that a good latent structure can lead to good transition matrix estimation. More details can be found in the A9 of the rebuttal.

Thanks again for your valuable comments. If there are any remaining concerns, we can discuss them.

Best regards,

Authors

Thanks authors for their sincere answers. Although authors have shown through efforts for their studies, I am still not convinced for the following points.

• I think part-dependent label noise is rather an old and specific version of instance dependent transition matrix studies. Rather, I think there are no manual similarity assumptions also in BLTM[1].

• After all, I think the motivation of why we should use the paper's method to estimate the transition matrix is mainly based on its performance. However, I am not convinced with its utility, since the performance gap does not seem so much (as reviewer afzG also pointed out)

• Furthermore, there should be some empirical findings showing the learned latent causal structure is optimized. If there are other factors that can affect the test accuracy as the authors said in the rebuttal, test accuracy cannot be an adjust metric showing the proposed method does capture the true transition matrix or true causal structure well or not.

• I doubt whether the baselines are well reproduced or not. For example, BLTM[1] performance, it should show better performance for instance dependent noise than the naive cross entropy, since it is the method mainly targeting the instance dependent noise. However, according to the authors' experiment results, it shows even worse experiment result for CIFAR10 instance dependent label noise setting than the naive cross entropy.

• Similar patterns also happen for many baselines including Mentornet, PTD, CausalNL, MEIDTM, BLTM, NPC, and RENT (considering CIFAR10 result). I am not sure what those papers' assumptions are for estimating the transition matrix, but I don't think the similarity assumption of each paper is the problem because the methods including Reweight, Forward and CCR shows better performances than CE as we expected.

Therefore, I will keep my initial score.

[1] Yang, S., Yang, E., Han, B., Liu, Y., Xu, M., Niu, G., & Liu, T. (2022, June). Estimating instance-dependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning (pp. 25302-25312). PMLR.

Thanks for your reply. The experimental results are well reproduced. We have carefully compared our results and the original papers’ results and the performance of baselines are comparable. We would like to provide the details for each baseline we used as follows.

For PTD and CausalNL, they also run experiments on instance-dependent label noise. Our experiment results are identical to their results.

Some baselines do not experiment on the instance-dependent label noise. Thus, we reproduce results on instance-dependent label noise. Our results are comparable to the results reported in existing papers. Specifically,

• CE, the standard cross-entropy loss. We reproduce it with PreAct-ResNet-18. A paper [1] also reproduce the results, the accuracies are 52.19±1.42 and 42.26±1.29 on CIFAR-100 under noise rates of 0.2 and 0.4. Our corresponding results are 54.98 ± 0.19 and 43.65 ± 0.15, which are comparable with the paper [1].

• MentorNet pretrains a classification network to select reliable examples for the main classification network. The original paper does not experiment on the instance-dependent label noise. We replace the CNNs used in MentorNet with PreAct-ResNet-18 and reproduce the results. A paper [2] also reproduces the results, the accuracies are 51.73±0.17 and 40.90±0.45 on CIFAR-100 under noise rates of 0.2 and 0.4. Our corresponding results are 55.98 ± 0.32 and 43.79 ± 0.48, which are comparable with the paper [2].

• Coteaching uses two classification networks to select reliable examples for each other. The original paper does not experiment on the instance-dependent label noise. We replace the CNNs used in Coteaching with PreAct-ResNet-18 and reproduce the results. A paper [1] also reproduces the results, the accuracies are 57.24±0.69 and 45.69±0.99 on CIFAR-100 under noise rates of 0.2 and 0.4. Our corresponding results are 61.54 ± 0.06 and 49.50 ± 0.10, which are comparable with the paper [1].

• Forward uses the transition matrix to correct the loss function. The original paper does not experiment on the instance-dependent label noise. We replace the ResNets used in Forward with PreAct-ResNet-18 and reproduce the results. A paper [1] also reproduces the results, the accuracies are 58.76±0.66 and 44.50±0.72 on CIFAR-100 under noise rates of 0.2 and 0.4. Our corresponding results are 56.59 ± 0.25 and 46.03 ± 0.65, which are comparable with the paper [1].

• DivideMix divides the noisy examples into labeled examples and unlabeled examples and train the classification network using semi-supervised technique MixMatch. The original paper does not experiment on the instance-dependent label noise. We follow their original settings to reproduce the results. A paper [3] also reproduce the results, the accuracies are 77.07 and 70.80 on CIFAR-100 under noise rates of 0.2 and 0.4. Our corresponding results are 76.81 ± 0.14 and 73.12 ± 0.32, which are comparable with the paper [3].

There are two baselines, CCR and Reweight, modeling class-dependent label noise; the original paper does not experiment on the instance-dependent label noise, and we have not found related papers that report their performance on the instance-dependent label noise. Specifically,

• CCR uses forward-backward cycle-consistency regularization to learn noise transition matrices. Their original paper does not experiment on the instance-dependent label noise. We use a PreAct-ResNet-18 as the backbone and follow their original settings to reproduce the results.

• Reweight estimates an unbiased risk defined on clean data using noisy data by using the importance reweighting method. Their original paper does not experiment on the instance-dependent label noise. We use a PreAct-ResNet-18 as the backbone and follow their original settings to reproduce the results.

For BLTM and MEIDTM, our results are different from those of their paper as we use the instance-dependent label noise generation method in PTD [4]. The results of their paper used different instance-dependent label noise generation methods. Specifically,

• For BLTM, they assume the noise rates have upper bounds. As stated in the Problem setting section of their paper [5]: “This paper focuses on a reasonable IDN setting that the noise rates have upper bounds $\rho_{max}$ as in (Cheng et al., 2020)”. To generate such instance-dependent label noise, they modify the instance-dependent label noise in PTD [4]. As shown in the second line of the Algorithm 2 in [5]: “Sample instance flip rates $q_i$ from the truncated normal distribution $\mathcal{N}(\eta,0.1^2,[0,\rho_{max}])$, which is different from the original algorithm in PTD [4]: “Sample instance flip rates $q\in \mathbb{R}^n$ from the truncated normal distribution $\mathcal{N}(\tau,0.1^2,[0,1])$;”, i.e., The truncated normal distributions in two algorithms are different.

• For MEIDTM, they follow the instance-dependent label noise generation method in the work [6]. This instance-dependent label noise generation method is also different from the method in PTD. The Line 3 of Algorithm 1 of this paper [6] is “Sample $W \in \mathcal{R}^{S \times K}$ from the standard normal distribution $\mathcal{N}(0,0.1^2)$;”, while the original step in PTD is “Independently sample $w_1,w_2,\dots,w_c$ from the standard normal distribution $\mathcal{N}(0,0.1^2)$;”. Correspondingly, the Line 4 of the Algorithm 1 of the paper [6] is “$p=x_n\cdot W$”, while the original step in PTD is “$p=x_i\cdot w_{y_i}$”. The original noise generation algorithm has $c$ parameters $w_i$, where $c$ is the number of the class, but the noise generation method in MEIDTM only has one parameter $W$.

Reference

[1] Bai, Yingbin, et al. "Understanding and improving early stopping for learning with noisy labels." Advances in Neural Information Processing Systems 34 (2021): 24392-24403.

[2] Yuan, Suqin, Lei Feng, and Tongliang Liu. "Late stopping: Avoiding confidently learning from mislabeled examples." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Garg, Arpit, et al. "Instance-dependent noisy label learning via graphical modelling." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2023.

[4] Xia, Xiaobo, et al. "Part-dependent label noise: Towards instance-dependent label noise." Advances in Neural Information Processing Systems 33 (2020): 7597-7610.

[5] Yang, Shuo, et al. “Estimating Instance-dependent Bayes-label Transition Matrix using a Deep Neural Network.” arXiv preprint arXiv:2105.13001 (2021).

[6] Cheng, Hao, et al. "Learning with instance-dependent label noise: A sample sieve approach." arXiv preprint arXiv:2010.02347 (2020).

Dear Reviewer pfn2,

Thank you very much for your time and comments. To further clarify our paper, we will follow your valuable advice and include the aforementioned details in our revision. We hope this can address your concern well.

The author-reviewer discussion ends soon, please kindly let us know if there are any additional concerns or suggestions. We are happy to provide answers.

Many thanks,

Authors

Response to Reviewer afzG

Q1: I don't get why latent factors generating $\mathbf{X}$ and latent factors generating noisy label $\tilde{Y}$ have to be different? And how to enforce it?

A1: Thank you for your question. The latent factors for generating instance $X$ and noisy label $\tilde{Y}$ can be the same, as our method is flexible enough to allow the masks to select any subset of latent factors, including potentially all factors, for generating $X$ and $\tilde{Y}$.

We introduce the masks to select factors because not all latent features $X$ cause $\tilde{Y}$. For instance, in Figure 1(a), the feature "fur" causes the noisy label "dog," whereas other features like "wall" and "floor" are irrelevant to the noisy label. To effectively isolate the minimal necessary latent factors for generating noisy labels, we employ an L1 norm constraint on the mask variables.

To empirically validate our method, we conducted an experiment on "moon dataset". More settings about the "moon dataset" can be found in the global response. After training, the values of the mask variable $M_{\tilde{Y}}$ for noisy labels are [6.0166e-05, 2.3246e-02], which means that the second factor is selected. These results demonstrate that our mask mechanism can effectively select the latent factor for generating noisy labels.

Q2: Performance between the best and second best method are less than 1% in several cases on CIFAR-10N.

A2: The CIFAR-10 is a relatively easy dataset where models can achieve high accuracy. For instance, the PreAct-ResNet-18 achieves an accuracy of 93.17 ± 0.04% when trained on clean data. However, our model demonstrates significant effectiveness on more challenging datasets with limited samples, such as CIFAR100N. Here, our method achieves an accuracy of 74.60 ± 0.17%, outperforming DivideMix, whose accuracy is 61.54 ± 0.34%. Moreover, our model also shows robust performance on the large-scale real-world dataset WebVision, achieving a top-1 accuracy of 79.84%, compared to 77.32% for DivideMix.

Q3: How significant is the step of selecting clean samples? Can we have another baseline of training classifier only over these clean samples?

A3: Thank you for your question. The selection of clean samples is a crucial step in many noise-robust algorithms, including CausalNL, InstanceGM, and DivideMix, which use a heuristic approach of alternatively training classifiers and selecting clean samples. The accuracy of the selected clean samples improves as the improvement of the classifier. Our method follows the heuristic process.

We train a baseline using cross-entropy loss on the selected clean samples. We use "CE-clean" to denote this baseline. The experiment results are in Table 5 and 6 of the PDF.

Q4: The optimization problem is complicated as it serves as a criterion to learn both casual model structure and parameters.

A4: Though the optimization problem is complicated, the optimization process of our model is stable using modern optimizers, such as SGD and Adam.

Q5: I would suggest to have more discussion on the identifiability result.

A5: Thanks for your advice. Here, we provide more discussion on the identifiability result: The theoretical results indicate that when the number of causal factors is 4, $n_s*15$ confident examples from distinct classes are required to identify the causal model, where $n_s$ is the number of different style combinations. If the changes in style combinations do not affect the parameters of our causal model, only 15 confident examples from distinct classes are required. This discussion will be added to the final version of our paper.

Q6: What is the number of latent factors $Z_i$?

A6: Thank you for your suggestion. The number of latent factors is 4, as mentioned in Line 632 of our paper. To clarify this and prevent any confusion, we will add a statement in Section 4 of the final version of the paper to emphasize that the number of latent factors is 4.

Q7: What are $g_Y^1,g_Y^2$ in the Algorithm 1?

A7: $g_Y^1$ and $g_Y^2$ are two classification networks used to model the distribution of $q_\psi(Y|X)$, which are defined in line 197 of the paper. Specifically, we follow previous work (DivideMix) to adopt a Co-Teaching learning paradigm. We will clarify this in Appendix D.

Q8: How is $q_{\psi}$ updated after the warm-up step?

A8: After the warm-up step, the entire model, including $q_{\psi}$, is optimized end-to-end by minimizing the loss defined in Equation 9.

Q9: How the mask variables $M_X,M_{\tilde{Y}}$ are updated? Or are they considered as variables?

A9: The mask variables $M_X$ and $M_{\tilde{Y}}$ are learnable parameters that are updated through the optimization of Equation 9. During this optimization, two key constraints apply to these variables: First, the causal factors selected by $M_X$ and $M_{\tilde{Y}}$ can be used to generate $X$ and $\tilde{Y}$; Second, the mask variables are required to be sparse. These constraints are integral to the loss defined in Equation 9. We will clarify the update of mask variables in the final version.

Q10: How many clean samples have been selected in the experiment? How accurate are they?

A10: Thanks for your insightful question. We report the number and the accuracy of the selected clean samples on the CIFAR-10 and CIFAR-100 datasets, as shown in Table 7 and 8 of the PDF. We will report the number and the accuracy of the selected clean samples in the Appendix of our paper.

Q11: Under the proposed casual model, can we have $P(\tilde{Y}|Y,X)=P(\tilde{Y}|Y)P(Y|X)$ ? I suggest to have such discussion to clearly distinguish the proposed model vs the instance-independent confusion matrix.

A11: Thanks for your insightful question. We believe you mean to $P(\tilde{Y}|Y,X) = P(\tilde{Y}|Y)$. Our model is instance-dependent (see details on the global response). We will clarify this distinction in the introduction of our paper.

Dear authors,

Thank you for addressing my concerns/questions. I have a few more questions.

Q1. In the paper, it is stated clearly that they are different. Does that mean the method allows the latent factors generating instance and noisy label to be different, but they do not have to (or should not?) be in general?

Q6. Why it is 4 in particular? Does 4 play a crucial role in result, theoretically or empirically?

Q7, 8, 9. I am really confused. Why are there two classifiers $g_Y^1, g_Y^2$? What are the difference between them?

In the Algorithm1, $g_Y^1, g_Y^2$ are output of WarmUP step, but the Warmup section in page 5 never mentions about the 2 classifiers. In particular, in line 197, there is only 1 "classification network $g_Y$" which is used to "model the distribution $q_\psi (Y |X)$".

Also, is $g_Y$ or the $q_\psi (Y |X)$, or both trainable?

Q9. The term mask is usually used to imply binary vectors. Are those mask vectors $M_X, M_Y$ not binary vectors? Do they have any constraints beside sparsity?

Response to Reviewer S6tG

Q1: There is a lack of analysis on whether the model in Figure 2(b) is more effective than the model in Figure 2(a). To better demonstrate the effectiveness of the proposed model, I recommend either reporting the performance of InstanceGM or describing the results of applying various classifier training methods instead of relying solely on MixMatch.

A1: Thank you for your suggestion. We have conducted experiments to assess whether the model in Figure 2(b) outperforms the model in Figure 2(a). Specifically, we replaced the generative models in CausalNL and InstanceGM (as shown in Figure 2(a)) with our proposed model (Figure 2(b)), while maintaining identical settings for the other experimental parameters. These experiments were conducted on the CIFAR-10 and CIFAR-100 datasets, which feature instance-dependent label noise. We refer to the modified versions of CausalNL and InstanceGM as "CausalNL'" and "InstanceGM'", respectively. The results, presented in Tables 3 and 4 of the PDF, indicate that the model in Figure 2(b) is indeed more effective than the model in Figure 2(a). We will include these results in the final version of our paper.