Incorporating Implicit Regularization to Enhance the Transition Matrix Method for Effective Handling of Diverse Label Noise

$**For writing: **$

The first point:

We have provided detailed definitions of X and Y in preliminaries (Section 2.1). However, we acknowledge your suggestion that it would be appropriate to provide them in the initial introduction part as well.

The second point:

Intuitively, our model primarily utilizes the backbone network as main framework without employing time-consuming techniques such as sample selection, contrastive learning for feature training, or graph model construction. To illustrate, we conducted tests on the CIFAR-100 dataset to record the running time for some methods mentioned in our paper with a single NVIDIA 3090Ti, and the results are as follows. We observed that our method does not significantly increase the time consumption compared to those previous transition matrix methods for class-dependent noise. In contrast, approaches that are designed to handle instance-dependent noise or methods with more complex model exhibit higher time consumption.

The third point:

The symbol $-\top$ denotes the transpose of the inverse matrix, i.e., what you understand "Inverse + Transpose". We should provide a specific explanation of this notation in the paper. Thank you for pointing out this issue.

The fourth point:

The form of $u \odot u-v \odot v$ have served as an implicit regularization under specific conditions (see Appendix B.2). It is a technique that has been widely used before, e.g., [6], [7] and [3]. This approach can approximate the effect of constraining the L1 norm to achieve sparsity without the need for an additional regularization term and easier to optimize training than directly use L1 regularization. You can have a certain understanding through these articles and a detailed introduction about it can be seen in Appendix A.2 in our paper.

The fifth point:

Our method is a plug-and-play model, which can be combined with any other SOTA technologies, such as semi-supervised, or can be used as a downstream task of contrastive learning, to obtain better results. This does not conflict. However, the core purpose of our paper is to build a relatively simple framework that can achieve good results on various types of noise data. What we pursue is the simplicity of transition matrix methods and the theoretical analysis under certain assumptions. Therefore, we only compare our method with the same type of works, and the combination with other relatively complex SOTA technologies is not the focus of this paper.

Once again, we sincerely appreciate your patient review and suggestions. We hope that the additional explanations above can address your concerns and receive your approval.

[1] Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning, pp. 25302–25312. PMLR, 2022.

[2] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label noise learning without anchor points. In International conference on machine learning, pp. 6403– 6413. PMLR, 2021b.

[3] Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over parameterization. In International Conference on Machine Learning, pp. 14153–14172. PMLR, 2022.

[4] Bartlett, P. L., & Mendelson, S. (2002). Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov), 463-482.

[5] Mendelson, S. (2014). Learning without concentration. Journal of Machine Learning Research, 15(1), 589-613.

[6] Peng Zhao, Yun Yang, and Qiao-Chu He. Implicit regularization via hadamard product over parametrization in high-dimensional linear regression. arXiv preprint arXiv:1903.09367, 2(4): 8, 2019.

[7] Jiangyuan Li, Thanh Nguyen, Chinmay Hegde, and Ka Wai Wong. Implicit sparse regularization: The impact of depth and early stopping. Advances in Neural Information Processing Systems, 34: 28298–28309, 2021a.

Dear reviewer,

Thank you for your review of our paper and providing constructive questions and suggestions. We will address each of the questions you raised and are happy to continue further discussion on related issues with you.

$**For contributions:**$

The first point:

Specifically, our method has theoretical analysis results for the specific noise patterns you mentioned. However, it is a misunderstanding that our model is not applicable to other instance-dependent noise. The main objective of our paper is to develop a lightweight and practical model that performs well on a broader range of data. Our experimental results indicate that even on more complex instance-dependent data and real-world datasets, our method is more effective compared to other methods of the same type. This demonstrates that despite not fully conforming to our model assumptions, our model is still effective in fitting and learning. (Similar to that you can fit all data with a linear model, whether the effect is appropriate is the key.) Our model exhibits good results for various types of noisy datasets without the need for additional complex structures or techniques, which we believe is meaningful.

The second point:

We have previously read the work you mentioned, which focuses specifically on the transition matrix method in the context of multiple annotators. However, in our considered problem, each sample has only one label, so the research questions and the datasets used in our work are different. Therefore, we did not engage in a comparative discussion with that particular method. The baseline method of this method is TMDNN ([1]), it is an instance-dependent method. We have conducted a detailed comparison between our method and TMDNN in our paper, and our method outperforms it to a great extent.

$**For theoretical analysis: **$

The first point:

Our method primarily leverages analytical approaches concerning transition matrix and implicit regularization. For a better understanding of the analytical framework and some conclusions, we recommend referring to articles [2] and [3]. These articles provide valuable insights and can enhance comprehension of our analysis approach.

The second point:

The term $\epsilon$ can be arbitrarily small here, as guaranteed by the convergence of Theorem 1. Therefore, the $\epsilon$ term on the right-hand side of the inequality in Theorem 2 can be controlled. If your question is about the tightness of the $\epsilon$-cover control, a more rigorous result can be obtained using Rademacher Complexity [4] or Dudley's inequality [5] from high-dimensional probability theory. This derivation would be relatively complex, and we can provide it in the formal version of the paper.

The third point:

What you said is correct that network parameters are not typically initialized from zero. However, the condition we propose in this context is derived following the simplification of the model analysis, similar to the simplification from Equation (9) to Equation (10). In reality, any initialization is valid, it does not affect our analysis results albeit with a constant term difference. We made this simplification for the sake of ease in derivation and writing, without the requirement limited to this condition in practical application.

Our definition is followed by the definition provided in [5], which has been more commonly used within label noise learning differs from the mainstream definition in matrix analysis. However, your suggestion is valid, and ambiguous definitions should be eliminated. We will remove this usage from our paper. Thank you for pointing that out.

In fact, the experimental results clearly demonstrate the significant improvement of our method compared to the baseline methods across various datasets. We are not quite clear about what you mean by "similar” effects. If you are referring to the comparison with the SOP method (which also utilizes implicit regularization), it is true that the improvement may not be evident at low noise rates. However, our method outperforms SOP noticeably at high noise rates, as we have extensively analyzed and discussed in our paper.

Once again, we sincerely appreciate your patient review and suggestions. We hope that the additional explanations above can address your concerns and receive your approval.

References

[1] Zhou, Xiong, et al. "Learning with noisy labels via sparse regularization." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[2] Liu, Sheng, et al. "Robust training under label noise by over-parameterization." International Conference on Machine Learning. PMLR, 2022.

[3] Peng Zhao, Yun Yang, and Qiao-Chu He. Implicit regularization via hadamard product over parametrization in high-dimensional linear regression. arXiv preprint arXiv:1903.09367, 2(4): 8, 2019.

[4] Jiangyuan Li, Thanh Nguyen, Chinmay Hegde, and Ka Wai Wong. Implicit sparse regularization: The impact of depth and early stopping. Advances in Neural Information Processing Systems, 34: 28298–28309, 2021a.

[5] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label noise learning without anchor points. In International conference on machine learning, pp. 6403– 6413. PMLR, 2021b.

Dear reviewer,

Thank you for your review of our paper and providing constructive questions and suggestions. We will address each of the questions you raised and are happy to continue further discussion on related issues with you.

For Weaknesses:

Our method and the T-revision are two completely different approaches. T-revision adjusts the estimation of the transition matrix $\boldsymbol{T}$ by $\boldsymbol{T}+\delta \boldsymbol{T}$, which essentially constructs a transition matrix (independent of feature X) to handle class-dependent noise problems without utilizing the information of feature X to $\boldsymbol{T}$. Therefore, it is not suitable for instance-dependent noise problems. In contrast, our method directly fits the residual term, which contains information from X, rather than directly utilizing information from $P(\boldsymbol{Y} \mid X)$. Thus, our method is not limited to class-dependent noise problems and can be extended to instance-dependent noise. If, as you mentioned, representing the form $\gamma(X)=(\delta\boldsymbol{T})^{\top} P(\boldsymbol{Y} \mid X)$, where $\delta\boldsymbol{T}$ is independent of X, and $P(\boldsymbol{Y} \mid X)$ is unknown and need to be estimated by networks. In this case, adding $\gamma(X)$ will not provide extra improvement for learning $\boldsymbol{T}$ and $P(\boldsymbol{Y} \mid X)$ compared to directly using the transition matrix. Experimental results also demonstrate that our method substantially better than T-revision on various datasets, especially in instance-dependent noise scenarios, indicating the fundamental differences between these two methods.

We would like to introduce the origin of our ideas here, which may help you understand our method. Our core idea is to construct a lightweight framework based on the transition matrix method, which can transition from solving class-dependent noise to solving instance-dependent noise. Since it is difficult to learn the matrix $\boldsymbol{T}(X)$ (depend on X) for handling instance-dependent noise, we hope to combine the transition matrix and implicit regularization techniques as substitute. We aim to estimate an overall transition matrix similar to the previous class-dependent methods, but it is not enough to handle various complex types of noise. However, on the other hand, after the real label distribution $P(\boldsymbol{Y} \mid X)$ undergoes an overall transfer by the transition matrix $\boldsymbol{T}$, the difference between $\boldsymbol{T}^{\top} P(\boldsymbol{Y} \mid X)$ and the noise label distribution $P(\tilde{\boldsymbol{Y}} \mid X)$ should not be large. Therefore, we have the idea of through constraining the norm of the residual term to construct the model. Inspired by the idea from previous work using sparsity to deal with noise problems (e.g. [1], [2]), a sparse structure is used to make the residuals relatively small, which leads to our method TMR and we find it is widely effective. In fact, we have also tried to restrict the L2 norm of the residual term, but the results were not as good as now. If the observed noisy label data can be regarded as generated in this way (through a transition matrix and sparse residual term), we can theoretically demonstrate the effectiveness of our method (Theorem 1). In practical applications, experiments have shown that many instance-dependent noise data or real-world datasets can be considered as approximations of our assumption due to our method performs better than these specially designed instance-dependent transition matrix methods. This means that although the assumption of sparse residual may not always hold, the general fit of our framework for various types of noisy label data is valuable.

The form of $u_i \odot u_i-v_i \odot v_i$ have already served as an implicit regularization under specific conditions (see Appendix B.2). It is a technique that has been widely used before (e.g., [2], [3], [4], detailed introduction about it can be seen in Appendix A.2). This approach can approximate the effect of constraining the L1 norm without the need for an additional regularization term and easier to optimize training than directly use L1 regularization, hence it called "implicit" regularization. In Theorem 1 of our paper, we have demonstrated the identifiability of $u_i$, $v_i$, the transition matrix $\boldsymbol{T}$ and network parameters $\theta$ under specific conditions.

In our paper, the term "sample" is used to denote random instance from dataset that lack referential meaning and do not require a specific selection process. Does this align with your understanding of the term "example"? In reality, we have confusion in the interchangeable usage of these two terms in related papers within the field. Could you provide clarification on the distinctions between them? If indeed a misuse exists, we will modify it.

Dear reviewer,

Thank you for your review of our paper and providing constructive questions and suggestions. We will address each of the questions you raised and are happy to continue further discussion on related issues with you.

For Weaknesses:

The residual term refers to the difference between the true posterior probability of the noise data $P(\tilde{\boldsymbol{Y}} \mid X)$ and the probability distribution after multiplication by the transition matrix $\boldsymbol{T}^{\top} P(\boldsymbol{Y} \mid X)$. In fact, this residual term is not necessarily sparse, but we use a sparse structure to fit this residual term in the paper. The idea to use sparse residual term comes from previous work using sparsity to deal with noise problems (e.g. [1], [2]). We hope to fit the noisy label through a suitable transition matrix for the overall data combined with this sparse term. Theorem 1 is to illustrate that when the source of noise data can be regarded as or approximated as being generated in this way, our method can theoretically achieve optimal learning effects. In the experimental part, the experimental results on various types of noisy label data show that our method is universal, and it outperforms other existing transition matrix methods, regardless of class-dependent label noise, instance-dependent label noise, or unknown real-world datasets. (This does not mean that our model assumptions are always correct, but compared to other methods, it has a better effect of fitting noisy label data.)

The first point:

As mentioned above, we aim use experiments to show that our proposed method is not only effective under the assumption of our model, but also universally applicable to various types of noisy data and exhibits high practical reliability. Moreover, the experimental results indicate that compared to other transition matrix methods, we can outperform those specifically designed methods for specific type of label noise on corresponding datasets.

The second point:

The “valid” transition matrix and sparse residual term refer to our desire to fit the noisy label distribution $P(\tilde{\boldsymbol{Y}} \mid X)$ using a model that multiplies a transition matrix $\boldsymbol{T}$ with the clean label distribution $P(\boldsymbol{Y} \mid X)$ (learned through the network) and adds a sparse term $\gamma(X)$. If the observed noisy label data is indeed generated or can be approximately viewed in this way, we have conducted analysis in the theoretical section. Theorem 1 provides the conclusion that our method can learn network parameters that are statistically consistent with the parameters learned with clean label data, which means the ability to learn clean label distributions through network. In other unknown situations, there is no direct theoretical analysis results for our model, but this does not mean that our method is limited. Experimental results show that our method performs better than existing transition matrix methods on either synthetic instance-dependent noisy data or real-world noisy datasets. Therefore, our model is effective in practice. (That means our model may not be accurate for all noisy label data, but it is universally effective and concise.)

The third point:

This is because this part $\log det(\boldsymbol{T})$ does not vary with the sample X, so its impact on both sides of the inequality for equation (21) and equation (22) are same. In fact, this assumption in the text is made for simplifying the loss function in the formula derivation and adding it up will not affect the result.

Thank you for your suggestion. In reality, we have already read those papers as you mentioned before writing our paper. However, they all use more complex model structures such as contrastive learning and graph models. The original idea of our paper is to propose a lightweight unified model framework that is more widely applicable to various types of label noise, based on the theoretical analysis of transition matrix methods. Therefore, we did not compare our work with those SOTA works because we are on different tracks. We specifically compared it within the works based on transition matrix. It is also worth mentioning that our method can be well integrated with those SOTA techniques, such as downstream tasks in contrastive learning, but this is not the focus of our paper.

If only considering the form of the loss function, our method does look like a combination of VolMinNet[3] and SOP[2]. However, it is not simply a patchwork. Our core idea is to propose a relatively concise unified approach framework that can perform well on a wider range of noisy datasets. The analytical ideas are presented in the methods section. We are the first to propose the idea of combination transition matrix and implicit regularization to deal with the label noise problem, which can complement each other's advantages and overcome their respective shortcomings. Under certain assumptions, we have theoretically demonstrated the effectiveness of our method. Additionally, experimental results have shown that the method we proposed is a lightweight framework that outperforms previous methods of the same type on multiple types of noisy datasets.

We apologize for the oversight in our writing at the time. The links are highlight displayed during the editing process with the LaTeX compiler, lead to we neglect the generated PDF is difficult to read. We apologize for any inconvenience caused and will rectify this in the final version.

Once again, we sincerely appreciate your patient review and suggestions. We hope that the additional explanations above can address your concerns and receive your approval.

References

[1] Zhou, Xiong, et al. "Learning with noisy labels via sparse regularization." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[2] Liu, Sheng, et al. "Robust training under label noise by over-parameterization." International Conference on Machine Learning. PMLR, 2022.

[3] Li, Xuefeng, et al. "Provably end-to-end label-noise learning without anchor points." International conference on machine learning. PMLR, 2021.

Dear reviewer,

Thank you for your review of our paper and providing constructive questions and suggestions. We will address each of the questions you raised and are happy to continue further discussion on related issues with you.

For Weaknesses:

If only considering the form of the loss function, our method does look like a combination of VolMinNet and SOP. However, it is not simply a patchwork. Our core idea is to propose a relatively concise unified approach framework that can perform well on a wider range of noisy datasets. We are the first to propose the idea of combination transition matrix and implicit regularization to deal with the label noise problem, which can address the shortcomings of VolMinNet and SOP to obtain a model with a wider range of applicability. SOP can handle instance-dependent noise, but its performance is severely affected on datasets with high noise rates due to its requirement of noise sparsity. On the other hand, VolMinNet may have large errors when uses a fixed transition matrix to learn instance-dependent noise. Our proposed method (TMR) simultaneously mitigates the drawbacks of VolMinNet and SOP, resulting in a model with stronger fitting performance for more noisy datasets. Therefore, TMR is a meaningful job for label noise learning that can be considered as an extension of SOP for high noise rates, rather than an extension for instance-dependent angle.

Furthermore, we provide detailed analysis in both theory and experiments to demonstrate the effectiveness of our proposed method, constituting a complete work. It is a simple and widely applicable model that can be used as a plug-and-play method. As for the doubt regarding the similarity between our theoretical analysis and SOP, it is true that we draw inspiration from SOP's method analysis and use some results in our proof process for simplicity, which are mentioned in paper. However, the models are different and we combine the analysis of transition matrix part. The construction and analysis process of our model are innovative and not just a naive move from SOP. The complexity and analysis difficulty of the model are higher than SOP. Besides, we do not think that the reference to previous method’s proof should be considered as a lack of originality. Moreover, we also give analysis on generalization in Section 3.2, which is not included in SOP.

For Questions:

For Q1:

As you mentioned, CCR was proposed based on VolMinNet to alleviate the estimation error of noisy posterior probability. However, it is only applicable to class-dependent noise and does not work well for instance-dependent noise. Moreover, even for class-dependent noise, the framework of CCR is not as effective as our TMR, which can adjust the estimation of noisy posterior probability for each sample through implicit regularization. Taking the some results on CIFAR100 as an example, it can be observed that both for class-dependent noise data and instance-dependent noise data, there is a significant gap between CCR and our method.

For Q2:

Our initial idea is to use a relatively small residual term $\gamma(X)$ to fit the difference between $P(\tilde{\boldsymbol{Y}} \mid X)$ and $\boldsymbol{T}^{\top} P(\boldsymbol{Y} \mid X)$, and then use $\boldsymbol{T}^{\top} P(\boldsymbol{Y} \mid X)+ \gamma(X)$ to fit the noisy labels distribution $P(\tilde{\boldsymbol{Y}} \mid X)$. Choosing implicit regularization is based on its ability to achieve the effect of L1 regularization and it is easy to update and learn, inspired by method SOP. We have also tested to constrain L2 regularization and directly using L1 regularization in loss function before, but we failed to obtain the experimental results as good as the one mentioned in the paper.

For Q3:

In fact, we have experimented with ResNet34 on the CIFAR dataset and found that for our method, it performs slightly better than using ResNet18. However, our computing resources are limited. In order to simplify the experiments and reduce computational costs, we chose ResNet18. In terms of results, TMR with ResNet18 performs better than SOP with regardless of whether ResNet18 or ResNet34, especially as the noise rate increases. There should be no such concern for the Clothing1M and WebVision datasets, as both use the same underlying framework.

Once again, we sincerely appreciate your patient review and suggestions. We hope that the additional explanations above can address your concerns and receive your approval.