Learning a Reusable Meta Denoiser for Learning with Noisy Labels on Multiple Target Domains

1. This article has very limited innovation; it combines ideas from various fields such as meta-learning, learning with noisy labels, and domain adaptation, but it does not propose any meaningful techniques to integrate them effectively. Furthermore, the article is quite vague in its presentation. In the third section, it provides only a rough framework of the proposed method and lacks necessary discussions on algorithm details and potential issues that may arise. For example, it is not clear how the corrupted source data are generated in the proposed method.
2. In the training phase, this article directly trains a Denoiser on the noisy source data and applies it for the target data. The effectiveness of such process needs to be based on the following premises: First, there should exist some learnable connections between the source data and target data. Second, the user should be aware of the form of label noise in the target data and be able to artificially generate similar noise on the source data. However, the proposed method does not involve any domain adaptation aspects, nor does it make any assumptions about label noise. Besides, it also does not make much sense to extract features from two different and independent neural networks while apply only one denoiser.
3. The theoretical analysis in the article is directly derived from the classical generalization analysis based on the Rademacher complexity. It is unclear how this theory is related to the noisy label problem studied in the article, and it does not provide any assistance in theoretically understanding the proposed algorithm.

1. It would be better to clarify the contributions more clearly in Introduction.
2. How does the experiments conduct? Since the experiments involves random noises, it should be repeated several times. Are the experiments repeated? If so, in Tables 1 and 2, both the average results and the standard deviation should be reported. Especially, in Table 1, in some cases, the proposed methods perform very closely to some baseline methods.
3. The experimental results show that the performance of MeDe-L is very close to MeDe-C in most cases. If so, what is the necessaty for proposing these two methods?
4. On the real-world data sets, i.e., Table 3, the average reports and standard deviation are reported. However, the difference between the proposed ones and the compared baselines is so small. It seems that the difference is not statistically significant.

1. The use of binary classifier over meta features seems to be an excellent idea. However, the paper mentions that the top T (tau) data instances from the final probability list of target instances are considered. What is this threshold used in the evaluation? An ablation study regarding this threshold would be good to add. Furthermore, does this threshold depend on the performance of the binary classifier? These set of questions needs to be addressed and is completely missing in the paper.
2. The paper mentions that the authors tried multiple classifiers such as SVM, neural network etc. What worked and what did not work? Which specific classifier was used to obtain the results presented in Table 2, 3 and 4? What were their performance on the source domain used?
3. Since the solution assumes that the meta-label, i.e., whether the given class is noisy or not, is available, is this not additional effort over the source data labeling effort to create such meta-labels? Also, do you assume that these meta-labels are noise-free? What would happen if this meta-label itself is noisy?
4. As table 2 shows accuracy on symmetric and asymmetric label flipping, the accuracy scores across various competing methods seems close to each other. Is the difference statistically significant?

I believe that the authors should work on the following to improve their work:

1. Experimentation: There are several limitations in terms of the experiments that authors have conducted. I would encourage them to go through the recently proposed AQuA benchmark [1] to make their experimentation much more rigorous. Some of these limitations in decreasing order of importance are: (1) Recent baselines: MeDe is not compare with many recent state-of-the-art baselines such as AUM [3], Confident learning [4], SimiFeat [5], (2) Limited datasets: the authors only experiment on a limited number of datasets, missing out on widely-used datasets for noisy learning e.g. Clothing1M (or its subset Clothing100K), or datasets of practical significance such as that of noisy chest X-rays (NoisyCXR) [6]. I would also be more interested in seeing transfer from CIFAR to NoisyCXR, or CIFAR to Clothing100K, (3) Multiple runs: All results are reported over a single run, and hence it is not possible to compare the significance of differences, (4) Arbitrarily large noise rates: The authors run experiments with 40% noise, which in my opinion, is unrealistic, and much higher than the noise rate of real world datasets, (5) Base classification models: the authors only report their results on one base classification architecture, and not different architectures, (6) Methods to add label noise: The study uses a limited number of synthetic label noise generation methods, when many other ways to inject label noise exist, such as class-dependent label noise. Overall, I highly encourage the authors to follow and extend the evaluation protocol proposed in [1].
2. Premise: I am not convinced that training loss dynamics and confidence of trained models are indeed shared across datasets. Perhaps performing more experiments, on more datasets, as I suggested above might help demonstrate this. Also AUM and Confident learning use these very heuristics to identify labeling errors (and more), yet the authors fail to cite these papers.
3. Clarity: Some important details are missing from the paper. See Questions.
4. Theory: I don't think the theorem provided by the authors tells us anything different from what is already well known. Could the authors explain what is the new theoretical insight that they provide?

References: [1] Goswami, M., Sanil, V., Choudhry, A., Srinivasan, A., Udompanyawit, C., & Dubrawski, A. (2023). AQuA: A Benchmarking Tool for Label Quality Assessment. arXiv preprint arXiv:2306.09467.

[2] Chong, Derek, Jenny Hong, and Christopher D. Manning. "Detecting Label Errors using Pre-Trained Language Models." arXiv preprint arXiv:2205.12702 (2022).

[3] Pleiss, G., Zhang, T., Elenberg, E., & Weinberger, K. Q. (2020). Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33, 17044-17056.

[4] Northcutt, C., Jiang, L., & Chuang, I. (2021). Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70, 1373-1411.

[5] Zhu, Z., Dong, Z., & Liu, Y. (2022, June). Detecting corrupted labels without training a model to predict. In International conference on machine learning (pp. 27412-27427). PMLR.

[6] Bernhardt, M., Castro, D. C., Tanno, R., Schwaighofer, A., Tezcan, K. C., Monteiro, M., ... & Oktay, O. (2022). Active label cleaning for improved dataset quality under resource constraints. Nature communications, 13(1), 1161.