Learning the Latent Noisy Data Generative Process for Label-Noise Learning

We sincerely thank you for your time and effort in reviewing our manuscript and providing valuable comments. Please find the response to your comments below.

Q1: BLTM has no assumption for the similarity between noise transition for different instances, thus the study on BLTM contradicts the author's proposal.

A1: BLTM can learn the noise transition of some examples using a neural network. The authors of BLTM hope the network can generate the noise transition for unseen instances. However, the author does not discuss how to generalize the learned noise transition to other examples, and they do not provide a theoretical guarantee that the noise transition can be generalized. Some hidden assumptions must exist to build the connection. Otherwise, the noise transition can not be generalized to other examples. Our claim focuses on the methods that have theoretical guarantees on the generalization of noise transitions, e.g., the noise transition is class-dependent, and the noise transition for the instances in the same manifold is the same. To avoid confusion, we have modified the paper.

We are the first to explore when the learned noise transition in some instances can be generalized to other instances in learning with noisy labels. Our aim is to enable deep neural networks to capture the connections of noise transitions among different instances automatically. Since once the connections are captured, the noise transition learned in some examples can be generalized to other examples. To achieve this, we need to model the joint distribution of all variables $p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})$. The noise transition can be obtained through

$$p(\tilde{Y}|\mathbf{X},Y)=\frac{\int_{\mathbf{Z},\mathbf{N}}p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})d\mathbf{Z}d\mathbf{N}}{\int_{\tilde{Y},\mathbf{Z},\mathbf{N}}p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})d\tilde{Y}d\mathbf{Z}d\mathbf{N}}.$$

To obtain this joint distribution, we propose a principled way to model the whole data generative process. Specifically, we not only model the generative process from the latent variable $\mathbf{Z}$ to the instance $\mathbf{X}$ but also model and learn causal structure among different latent variables $\{Z_1,Z_2,\dots,Z_m\}$ from observed noisy data.

To achieve this, the generation of the variable $\mathbf{N}$ is modeled by a Gaussian distribution. The generation of the latent variable $\mathbf{Z}$ is modeled by a Structure Causal Model (SCM): $Z_i:=\mathbf{W}^T_i \mathbf{Z}+N_i$, where $\mathbf{W}$ is a matrix used to represent the association among the latent variables. Two generators are used to model the generation of the instance $\mathbf{X}$ and the noisy label $\tilde{Y}$, respectively.

Moreover, our paper has analyzed the identifiability of the proposed method theoretically in Appendix C. We discuss the conditions required for identifiability. Specifically, when the number of distinct classes is large (more than the sufficient statistics of the latent variable $\mathbf{Z}$) and we have a confident example in each class, the latent variable $\mathbf{Z}$ can be identified up to permutation.

Q2: The authors actually make another assumption, saying that the selected dataset is clean.

A2: We assume that the selected dataset is likely clean. This assumption is reasonable because the accuracy of the selected dataset is high empirically. For example, when the dataset is CIFAR-10, and the noise type is "worst", the accuracy of the selected dataset is 0.9568.

It is also worth mentioning that without clean labels, recovering the causal factor $\mathbf{Z}$ is an ambitious goal and generally impossible. This is because the nonlinear mixture function $f(\cdot)$ used to generate observed variable $\mathbf{X}$ is unknown. A variable $\mathbf{Z}'$ can be constructed via a nonlinear function $\mathbf{Z}'=g(\mathbf{Z})$. Another nonlinear mixture function $f'(\cdot)$ can generate the same observation, i.e., $\mathbf{X}=f'(\mathbf{Z}')$. Previous studies also prove that it is ill-posed to recover the latent causal factors with only the observed variables [1,2]. Existing methods in causal representation learning also have to use the auxiliary variables to recover the causal factors [3,4]. In the context of learning with noisy labels, auxiliary variables necessitate clean labels.

Q3: Time complexity and memory will be large.

A3: The proposed method does not increase the time complexity and memory because the encoder and the decoder introduced by our method are lightweight. The training time of the proposed method on CIFAR-10 is 13.68 hours, and the training time of DivideMix is 9.08 hours. The memory used for the proposed method is 7.06 GB, and the memory used for DivideMix is 6.99 GB.

Moreover, the encoder and decoder can be discarded during inference. The inference time is the same as the time of baselines.

Q8: How can we be sure that the process the authors proposed improves the model performance?

A8: Our method is conceptually better than other methods. As discussed in the Introduction, to generalize the noise transition learned in some instances to other instances, existing work without theoretical guarantee manually defines similarity for noise transition across various instances. However, the human-defined similarity is hard to verify in the real world. If the similarity is not truthful, the estimation error of the noise transition could be large. Thus, the performance of the classifier would decrease.

To avoid human-defined similarity, we propose a principled way to capture the connections of noise transitions among different instances automatically. Specifically, our method models the data generative process, and then the joint distribution $p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})$ can be obtained. Thus, the noise transition can be captured (please refer to the answer in A1). Once the model can capture the connections, the noise transition estimated in some examples can be generalized to other examples. The information contained in other examples can be inferred and learned by the model. Thus, the performance of the model will be improved. The experiment results also have verified the effectiveness of our method.

References

[1] Hyvärinen, Aapo, and Petteri Pajunen. "Nonlinear independent component analysis: Existence and uniqueness results." Neural networks 12.3 (1999): 429-439.

[2] Locatello, Francesco, et al. "Challenging common assumptions in the unsupervised learning of disentangled representations." international conference on machine learning. PMLR, 2019.

[3] Yang, Mengyue, et al. "Causalvae: Disentangled representation learning via neural structural causal models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[4] Liu, Yuhang, et al. "Identifying weight-variant latent causal models." arXiv preprint arXiv:2208.14153 (2022).

[5] Xia, Xiaobo, et al. "Part-dependent label noise: Towards instance-dependent label noise." Advances in Neural Information Processing Systems 33 (2020): 7597-7610.

[6] Xia, Xiaobo, et al. "Are anchor points really indispensable in label-noise learning?." Advances in neural information processing systems 32 (2019).

[7] Patrini, Giorgio, et al. "Making deep neural networks robust to label noise: A loss correction approach." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Q4: Whether the error of the inferred transition is small or not should be expressed.

A4: We have conducted experiments on the dataset CIFAR-10 to calculate the noise transition estimation error. The labels are corrupted manually using the instance-dependent noisy label generation method proposed in [5]. The baselines used to compare the noise transition estimation error are BLTM, MEIDTM and VolMinNet. The experiment results are shown as follows.

Empirical results show that our method can reduce the error of inferred noise transition.

Q5: Why should we use graph structure?

A5: To model the causal structure among latent variable $\mathbf{Z}$, the graph structure has to be used. This is because the causal structure among $\mathbf{Z}$ is a part of the noisy data generative process. As mentioned in A1, modeling the noisy data generative process can obtain the joint distribution $p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})$, then the connections of noise transitions among different instances can be captured automatically. The more accurately the generation process is modeled, the more accurate the joint distribution obtained will be.

Q6: Evidence of the proposal saying that "noise transitions can be inferred of a special group of instances"?

A6: The evidence is the noise transition estimation methods in previous work. For example, the special group of instances can be the anchor points in the dataset, which is stated in Section 2 in [6] and Theorem 3 in [7].

Q7: If the estimated labels are clean enough, why should we fundamentally do more processes?

A7: The size of selected examples is much smaller than the size of the whole dataset, and the distribution of selected examples is not the same as the distribution of examples in the clean domain. As a result, training classifiers on selected examples with different distributions will lead to the performance degradation of the classifiers because the parameters of the learned classifier are not statistically consistent with the parameters of the optimal classifier defined on the clean domain.

To guarantee the statistical consistency of classifiers, the information contained in the remaining examples has to be used. Modeling the noise transition for the whole dataset is a way to employ the information contained in the training examples. The reason is that the clean class posterior $p(\tilde{Y}|\mathbf{X}=\mathbf{x})$ for an instance $\mathbf{x}$ can be inferred through the noise transition $p(\tilde{Y}|Y,\mathbf{X}=\mathbf{x})$ and the noisy class posterior $p(\tilde{Y}|\mathbf{X}=\mathbf{x})$, i.e., $p(\tilde{Y}|\mathbf{X}=\mathbf{x})=p(\tilde{Y}|Y,\mathbf{X}=\mathbf{x})p(Y|\mathbf{X}=\mathbf{x})$.

Thank authors for their sincere efforts to respond to my concerns. Still, my concerns below are not solved yet, and I require the authors to respond to those.

1. I want to know the number of samples and accuracies with regard to the distilled dataset for several noisy dataset settings to support the proposal of authors, saying "assume the selected dataset is likely clean is reasonable" in A2 and "the distribution of selected examples is not the same as the distribution of examples in the clean domain" in A7. Also want to know the performance comparison when trained only with distilled dataset, since if the peformance only with the distilled dataset is similar to the performance the authors proposed in the paper, I think it will take far less time and memory complexity.

2. If the distribution of selected examples is not the same as the distribution of examples in the clean domain, how can the author can assume the learned causal factor with distilled samples is trustworthy?

3. Time and memory comparison is not enough. Dividemix uses two networks, so it is so certain that this method would spend computation than other baselines. The time and memory comparison with regard to all other baselines, or at least with regard to other transition matrix based methods can tell how much time and memory is used usually and enable the evaluation of the method.

Also, I think the time and memory gap could be larger when this method is applied to more complicated dataset, e.g. Clothing1M, than CIFAR10.

4. Can the authors specify how they calculated the transition matrix estimation error for the instance dependent transition matrix? e.g. how to measure the difference between the ground truth and the estimated transtion matrix, since the transition matrix should be different for every instance? / What is the meaning of estimation error of other dimensions when a true label of a sample is only one? e.g. the transition matrix error of label dimension 0,1, and 3~9 for 10-class classification when a true label of a sample is 2?

5. I found out the authors did not compare theirs with class-dependent transition matrix estimation based methods. However, I think it should be compared as baselines.

6. Additionally, I doubt the authors lack baselines. e.g. in [1] , which utilizes class dependent transition matrix, they achieved 75.12% for Clothing1M. However, this paper shows 74.81%, even with instance dependent transition.

7. I still cannot trust that the suggested method can significantly increae the resulting classifier's performance. I think this concern can be solved if the authors analyze what the graph structure tells about the relations between samples empirically, and how much different is the similarity inferred from the graph structure and human arbitrarily defined similarity (or currently modeled several instance-dependent transition matrix based methods).

[1] Cheng, D., Ning, Y., Wang, N., Gao, X., Yang, H., Du, Y., ... & Liu, T. (2022). Class-Dependent Label-Noise Learning with Cycle-Consistency Regularization. Advances in Neural Information Processing Systems, 35, 11104-11116.

Thank the authors again for their sincere efforts to relieve my concerns. I have questions below:

1. What is the reference [6] and [7]? I think there are only 5 references... If [7] is [7] Patrini, Giorgio, et al. "Making deep neural networks robust to label noise: A loss correction approach." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017., what is the difference between CCR and Forward?

2. I doubt experimental results are chosen since the dataset has changed from CIFAR10-IDN to CIFAR10-N from Q5. If the authors can explain why, I can understand.

3. According to various experimental results, I can compare at least BLTM, MEIDTM and GenP with regard to accuracy, transition matrix error. For accuracy, I found out BLTM<MEIDTM<GenP, while for transition matrix error, it is GenP<BLTM<MEIDTM. I want to say that the order of transition matrix error and accuracy is reversed at least for BLTM and MEIDTM. Is it right? Also, I want to see accuracy comparison with VolMinNet.

4. Is this method applicable to practical situations? For example, if we need 30 times more time (as the authors reported in the responses) for increasing 6% accuracy (as the authors reported in Clothing1M) than the naive cross entropy, is it practical?

5. It is just my curiosity (and it may not be important for the classification accuracy, but I'm not sure). Since GenP reconstructs the image again, can authors show how the image is generated? If the method worked correctly, the image should also have been generated correctly, right? I know that we do not have enough time to run the algorithm totally according to the reported time from the authors for this revision, but I suggest showing this generated image can be a way to show the algorithm has worked as the authors intended.

Reference

[1] Hyvärinen, Aapo, and Petteri Pajunen. "Nonlinear independent component analysis: Existence and uniqueness results." Neural networks 12.3 (1999): 429-439.

[2] Locatello, Francesco, et al. "Challenging common assumptions in the unsupervised learning of disentangled representations." international conference on machine learning. PMLR, 2019.

[3] Yang, Mengyue, et al. "Causalvae: Disentangled representation learning via neural structural causal models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[4] Liu, Yuhang, et al. "Identifying weight-variant latent causal models." arXiv preprint arXiv:2208.14153 (2022).

[5] Xia, Xiaobo, et al. "Part-dependent label noise: Towards instance-dependent label noise." Advances in Neural Information Processing Systems 33 (2020): 7597-7610.

[6] Xia, Xiaobo, et al. "Are anchor points really indispensable in label-noise learning?." Advances in neural information processing systems 32 (2019).

[7] Cheng, De, et al. "Class-Dependent Label-Noise Learning with Cycle-Consistency Regularization." Advances in Neural Information Processing Systems 35 (2022): 11104-11116.

Thanks the authors for answering all questions that I asked sincerely and sorry for my possible misunderstanding of the authors' intension.

Now I admit that the method of this paper makes instance dependent transition matrix well and it shows good performances for CIFAR-10 and CIFAR-10N dataset. The paper also shows more empirical analyses of why it works.

I still worry that the necessary previous baselines may not be compared enough (e.g. the authors could not state the reason of the superiority of CCR than their method for Clothing1M during the revision period) and question its applicability to practical settings, regarding time and computation complexity issue.

I hope the authors could reorganize including the analyses done during in this revision period well and upgrade their study and their manuscript for the next conference.

Currently, I will keep my score as before, but I think this paper is very interesting and promising.

We sincerely thank you for your time and effort in reviewing our manuscript and providing valuable comments. Please find the response to your comments below.

Q1: Highlight the differences from CausalNL in Related Work.

A1: Our method is different from CausalNL from several points.

• The aim of CausalNL is to exploit the information contained in the distribution of instances to help the learning of classifiers. To achieve this, the generative model is introduced. However, existing methods do not model the causal structure among latent variables. They only “partially” model the latent noisy data generative process without identifiablity analysis.

• Our aim is to enable deep neural networks to capture the connections of noise transitions among different instances automatically. Since once the connections are captured, the noise transition learned in some examples can be generalized to other examples. To achieve this, we propose a principled way to "fully" model the data generative process. Our method also models the causal structure among latent variables. We also analyzed sufficient conditions required for identifiability.

Here we would like to include more details. Due to limited space, we include the details in Appendix D to avoid confusion.

CausalNL helps the learning of classifiers by exploiting the information contained in the distribution of instances. To be specific, when the latent clean label $Y$ is a cause of the instance $\mathbf{X}$, the distribution of instance $p(\mathbf{X})$ will generally contain some information about the distribution of clean class posterior $p(Y|\mathbf{X})$. To exploit the information contained in the instances, CausalNL uses the generative model to model the generative relationship between the latent variable $\mathbf{Z}$ and the observed instance $\mathbf{X}$. However, they do not model the causal structure among the latent variable $\mathbf{Z}$. Therefore, their methods only “partially” model the noisy data generative process. These methods do not analyze the identifiability of the generative process.

Our aim is to enable deep neural networks to capture the connections of noise transitions among different instances automatically. Since once the connections are captured, the noise transition learned in some examples can be generalized to other examples. To achieve this, we need to model the joint distribution of all variables $p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})$. The noise transition can be obtained through

$$p(\tilde{Y}|\mathbf{X},Y)=\frac{\int_{\mathbf{Z},\mathbf{N}}p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})d\mathbf{Z}d\mathbf{N}}{\int_{\tilde{Y},\mathbf{Z},\mathbf{N}}p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})d\tilde{Y}d\mathbf{Z}d\mathbf{N}}.$$

To obtain this joint distribution, we propose a principled way to model the whole data generative process. Specifically, we not only model the generative process from the latent variable $\mathbf{Z}$ to the instance $\mathbf{X}$ but also model and learn causal structure among different latent variables $\{Z_1,Z_2,\dots,Z_m\}$ from observed noisy data.

Q2: Difference between the blue arrow and black arrow in Fig 2.

A2: Thanks for your suggestion. The black arrow indicates the causal direction in the structural causal model, while the blue arrow indicates the weights are changed across the clean label $Y$. More details about the generative process are included in Sec 3.

Q3: The assumption that the generation process of causal factors $\mathbf{Z}$ is linear should be given more details.

A3: Since causal factor $\mathbf{Z}$ is latent, discovering the causal structure is a challenging problem. If $\mathbf{Z}$ is observed, nonlinear causal mechanisms can be discovered easily. However, we do not assume the causal factors are observed. To provide a theoretical guarantee, we assume the causal mechanism among causal factors is linear. We empirically find that the performance of the proposed method is good, which indicates that this simplicity does not have negative effects.

Q4: Eq.(1) is very confusing.

A4: $\mathbf{W}$ is an upper triangular matrix with zero-value diagonal elements. When calculating $Z_i$, only the parents nodes of $Z_i$ are used, i.e., $Z_i=\sum_{j\in pa_i}W_{i,j}Z_{j}$.

Q5: Some related works are missing.

A5: Thanks for your advice. We have added these works to the Related Work.

Q6: What is the meaning of $N_i$.

A6: $N_i$ is an unexplained random variable that determines $Z$ in the structural causal model, i.e., $Z_i:=f_Z(pa(Z_i),N_i)$. This variable is referred as the noise variable in previous work [1], which is different from the noise in the label noise.

References

[1] Schölkopf, Bernhard, et al. "Toward causal representation learning." Proceedings of the IEEE 109.5 (2021): 612-634.

We sincerely thank you for your time and effort in reviewing our manuscript and providing valuable comments. Please find the response to your comments below.

Q1: Various studies have explored leveraging deep generative models to model complex data distributions and underlying noise patterns, thus using deep generative models (DGMs) to infer latent true labels is not a novel approach.

A1: Though some previous studies also use generative models, our work is different from previous work in several points.

• The aim of previous work CausalNL and InstanceGM is to exploit the information contained in the distribution of instances to help the learning of classifiers. To achieve this, the generative model is introduced. However, existing methods do not model the causal structure among latent variables. They only “partially” model the latent noisy data generative process without identifiablity analysis.

• Our aim is to enable deep neural networks to capture the connections of noise transitions among different instances automatically. Since once the connections are captured, the noise transition learned in some examples can be generalized to other examples. To achieve this, we propose a principled way to "fully" model the data generative process. Our method also models the causal structure among latent variables. We also analyzed sufficient conditions required for identifiability.

Here we would like to include more details. We have also included below in Appendix D to avoid confusion.

Previous work helps the learning of classifiers by exploiting the information contained in the distribution of instances. To be specific, when the latent clean label $Y$ is a cause of the instance $\mathbf{X}$, the distribution of instance $p(\mathbf{X})$ will generally contain some information about the distribution of clean class posterior $p(Y|\mathbf{X})$. To exploit the information contained in the instances, CausalNL and InstanceGM use the generative model to model the generative relationship between the latent variable $\mathbf{Z}$ and the observed instance $\mathbf{X}$. However, they do not model the causal structure among the latent variable $\mathbf{Z}$. Therefore, their methods only “partially” model the noisy data generative process. These methods do not analyze the identifiability of the generative process.

Our aim is to enable deep neural networks to capture the connections of noise transitions among different instances automatically. Since once the connections are captured, the noise transition learned in some examples can be generalized to other examples. To achieve this, we need to model the joint distribution of all variables $p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})$. The noise transition can be obtained through

$$p(\tilde{Y}|\mathbf{X},Y)=\frac{\int_{\mathbf{Z},\mathbf{N}}p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})d\mathbf{Z}d\mathbf{N}}{\int_{\tilde{Y},\mathbf{Z},\mathbf{N}}p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})d\tilde{Y}d\mathbf{Z}d\mathbf{N}}.$$

To obtain this joint distribution, we propose a principled way to model the whole data generative process. Specifically, we not only model the generative process from the latent variable $\mathbf{Z}$ to the instance $\mathbf{X}$ but also model and learn causal structure among different latent variables $\{Z_1,Z_2,\dots,Z_m\}$ from observed noisy data.

To achieve this, the generation of the variable $\mathbf{N}$ is modeled by a Gaussian distribution. The generation of the latent variable $\mathbf{Z}$ is modeled by a Structure Causal Model (SCM): $Z_i:=\mathbf{W}^T_i \mathbf{Z}+N_i$, where $\mathbf{W}$ is a matrix used to represent the association among the latent variables. Two generators are used to model the generation of the instance $\mathbf{X}$ and the noisy label $\tilde{Y}$, respectively.

Moreover, our paper has analyzed the identifiability of the proposed method theoretically in Appendix C. We discuss the conditions required for identifiability. Specifically, when the number of distinct classes is large (more than the sufficient statistics of the latent variable $\mathbf{Z}$) and we have a confident example in each class, the latent variable $\mathbf{Z}$ can be identified up to permutation.

Q2: The deep generative model introduces a significant computational burden.

A2: The proposed method does not increase the time complexity and memory because the encoder and the decoder introduced by our method are lightweight. The training time of the proposed method on CIFAR-10 is 13.68 hours, and the training time of DivideMix is 9.08 hours. The memory used for the proposed method is 7.06 GB, and the memory used for DivideMix is 6.99 GB.

Moreover, the inference time is the same as the time of baselines because the encoder and decoder can be discarded during inference.

Q2: Need to discuss more related work for learning with noisy labels.

A2: Thanks for your advice. We have added a paragraph in related work after introducing the noise-transition-based method to introduce other approaches, including sample-selection-based methods, other methods using generative models to facilitate learning with noisy labels, and the method leveraging the property of the label noise. Specifically, the paragraph is as follows.

Some noise-robust algorithms select examples deemed likely to be accurate for training purposes. These selections are based on the memorization effect, which suggests deep neural networks initially learn dominant patterns before progressively learning less common ones. In noisy label environments, accurate labels often constitute the majority, leading networks to prioritize learning from examples with accurate labels, typically indicated by lower loss values. Co-Teaching employs this principle to identify low-loss examples as likely accurate. DivideMix uses a Gaussian Mixture Model to separate training examples into labeled and unlabeled sets based on their training loss, with the labeled set presumed to contain accurate labels. Additionally, some methods use generative models to facilitate learning with noisy labels. CausalNL and InstanceGM utilize instance-specific information to enhance classifier learning. Conversely, NPC focuses on the generative process of estimated clean labels, not the noisy data generation, using generative models for label calibration. Finally, SOP applies the sparse property of the label noise, i.e., incorrect labels are the minority, to prevent models from overfitting to label noise.

Q3: Include experiments on various real-world datasets such as WebVision, ANIMAL-10N, and Mini-Imagenet.

A3: Thanks for your advice. We are still running experiments on WebVision. We will post the results as soon as possible.

Q4: An ablation study and sensitivity analysis.

A4: Thanks for your advice. For the ablation study, the method with only $\mathcal{L}\_{semi}$ is the same as DivideMix. Thus, the experiment results of the model only trained with $L_{semi}$ are the experiment results of DivideMix. For the sensitivity analysis, we have conducted the sensitivity analysis for $\lambda_{ELBO}$ and $\lambda_M$ on the CIFAR-10N dataset with noise type ''worst''. The experiment results are plotted as figures in Appendix E. We also list the experiment results as tables here:

We sincerely thank you for your time and effort in reviewing our manuscript and providing valuable comments. Please find the response to your comments below.

Q1: There is previous work that models the generative processes for label-noise learning.

A1: We are sorry for the confusion. We believe this is caused by a misunderstanding of noisy data generative process.

• The aim of previous work CausalNL and InstanceGM is to exploit the information contained in the distribution of instances to help the learning of classifiers. To achieve this, the generative model is introduced. However, existing methods do not model the causal structure among latent variables. They only “partially” model the latent noisy data generative process without identifiablity analysis.

• Our aim is to enable deep neural networks to capture the connections of noise transitions among different instances automatically. Since once the connections are captured, the noise transition learned in some examples can be generalized to other examples. To achieve this, we propose a principled way to "fully" model the data generative process. Our method also models the causal structure among latent variables. We also analyzed sufficient conditions required for identifiability.

• Note that NPC does not model the noisy data generative process but models the generative process of estimated clean labels.

Here we would like to include more details. We have also included below in Appendix D to avoid confusion.

Previous work helps the learning of classifiers by exploiting the information contained in the distribution of instances. To be specific, when the latent clean label $Y$ is a cause of the instance $\mathbf{X}$, the distribution of instance $p(\mathbf{X})$ will generally contain some information about the distribution of clean class posterior $p(Y|\mathbf{X})$. To exploit the information contained in the instances, CausalNL and InstanceGM use the generative model to model the generative relationship between the latent variable $\mathbf{Z}$ and the observed instance $\mathbf{X}$. However, they do not model the causal structure among the latent variable $\mathbf{Z}$. Therefore, their methods only “partially” model the noisy data generative process. These methods do not analyze the identifiability of the generative process.

Our aim is to enable deep neural networks to capture the connections of noise transitions among different instances automatically. Since once the connections are captured, the noise transition learned in some examples can be generalized to other examples. To achieve this, we need to model the joint distribution of all variables $p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})$. The noise transition can be obtained through

$$p(\tilde{Y}|\mathbf{X},Y)=\frac{\int_{\mathbf{Z},\mathbf{N}}p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})d\mathbf{Z}d\mathbf{N}}{\int_{\tilde{Y},\mathbf{Z},\mathbf{N}}p(\mathbf{X},\tilde{Y},Y,\mathbf{Z},\mathbf{N})d\tilde{Y}d\mathbf{Z}d\mathbf{N}}.$$

To obtain this joint distribution, we propose a principled way to model the whole data generative process. Specifically, we not only model the generative process from the latent variable $\mathbf{Z}$ to the instance $\mathbf{X}$ but also model and learn causal structure among different latent variables $\{Z_1,Z_2,\dots,Z_m\}$ from observed noisy data.

To achieve this, the generation of the variable $\mathbf{N}$ is modeled by a Gaussian distribution. The generation of the latent variable $\mathbf{Z}$ is modeled by a Structure Causal Model (SCM): $Z_i:=\mathbf{W}^T_i \mathbf{Z}+N_i$, where $\mathbf{W}$ is a matrix used to represent the association among the latent variables. Two generators are used to model the generation of the instance $\mathbf{X}$ and the noisy label $\tilde{Y}$, respectively.

Moreover, our paper has analyzed the identifiability of the proposed method theoretically in Appendix C. We discuss the conditions required for identifiability. Specifically, when the number of distinct classes is large (more than the sufficient statistics of the latent variable $\mathbf{Z}$) and we have a confident example in each class, the latent variable $\mathbf{Z}$ can be identified up to permutation.

3). A comparison between end-to-end learning and alternative learning approaches.

We conduct the experiments of the alternative learning approach, which optimizes $\mathcal{L}\_{semi}$ and $- \lambda\_{ELBO} ELBO +\lambda\_M (\| \mathbf{M\_X} \|_1 + \| \mathbf{M\_{\tilde{Y}}} \|\_1)$ alternatively. The dataset is CIFAR-10N.

The experiment results are shown as follows. Empirically, the performance of the end-to-end learning approach is better than the alternative learning approach.

4). An ablation study based on the number of latent variables.

We have conducted the experiments on CIFAR-10N dataset with the noise type "worst". The experiment results are in Appendix H. When the number of latent variables is around 4, the classification performance of the model reaches the peak.

Dear Reviewer 5FMB

Thank you for your valuable comments on improving our paper. We have added more experiments. These experiment results are added to Appendix E and H. Please let us know if there are any further concerns. Many Thanks.

1). The ablation study compares model training with only $\mathcal{L}_{semi}$ and with the entire loss. Here, we show the experiment results on CIFAR-10 with instance-dependent label noise. The experimental results indicate that with the increase of noise levels, the benefits of modeling the generative process become more significant (>2% on IDN-40%).

2). A sensitivity analysis regarding hyperparameters ($\lambda_{ELBO}$ and $\lambda_{M}$).

We have conducted the sensitivity analysis for $\lambda_{ELBO}$ and $\lambda_M$ on the CIFAR-10N dataset with noise type ''worst''. The experiment results are plotted as figures in Appendix E. The experiment results show that when the $\lambda_{ELBO}$ and $\lambda_M$ are around $0.01$, the test accuracy of the model is highest.