Federated Learning with Local Openset Noisy Labels

• Experiment:
  • How is data partitioned in an experiment? In federated learning, there are IID and non-IID partitions (the current discussion is based on ground truth labels/features), while in the paper it is not mentioned.
  • the centralized noisy label learning baselines used in this paper are LC, Co-teaching, T-revision, which were proposed several years ago. Could you please use some new baselines?
  • Also, authors try to avoid comparison with DivideMix by claiming that DivideMix consumes too much local computation. One suggestion is authors can use a single model version of DivideMix (mentioned as DivideMix without co-training in the original paper).
  • Authors can use [1] ELR as a baseline since it consumes less time compared with DivideMix, but has similar performance.
• Another suggestion is authors can try to experiment with sharing direct label distribution information (say, if it is allowed to share privacy label information to server), and use this performance as "upper-bound" for LDP used in this paper.
• Please use at least one latest FL noisy label method as the baseline, for example [3] FedCorr [4] RHFL
• openest noisy label in federated learning: actually, the definition of "openest federated noisy label" setting is almost the same as one of the settings proposed in [2] FedNoisy, though they may use different noise generation implementation as in authors' work.

[1] Liu, Sheng, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. "Early-learning regularization prevents memorization of noisy labels." Advances in neural information processing systems 33 (2020): 20331-20342. [2] Liang, Siqi, Jintao Huang, Dun Zeng, Junyuan Hong, Jiayu Zhou, and Zenglin Xu. "FedNoisy: Federated Noisy Label Learning Benchmark." arXiv preprint arXiv:2306.11650 (2023). [3] Xu, Jingyi, Zihan Chen, Tony QS Quek, and Kai Fong Ernest Chong. "Fedcorr: Multi-stage federated learning for label noise correction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10184-10193. 2022. [4] Fang, Xiuwen, and Mang Ye. "Robust federated learning with noisy and heterogeneous clients." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

1. According to the standard definition of 'openset' in machine learning, the term typically refers to a scenario where there are two distinct label sets: known labels and unknown labels. In this paper, such a clear distinction between known and unknown label sets is not provided. Instead, the paper focuses on addressing a different issue related to class imbalance, which is more characteristic of non-IID federated learning. Consequently, it is not entirely accurate to claim that this paper solves the 'openset noise' problem in federated learning. Furthermore, it remains unclear how the proposed methods would perform in addressing a genuine openset noise problem, as the server might struggle to initialize T_DP without knowledge of both known and unknown labels.

2. The stability of the results in Table 3, particularly when the differential privacy noise level (e) is set to 100, raises questions. It is important to provide a detailed explanation of why the performance remains largely unaffected at this noise level and how this stability aligns with the theoretical guarantees of the proposed approach.

3. A notable weakness in this work is the absence of a comparative analysis with existing studies that aim to address label noise in federated learning, such as FedCorr (CVPR 2022 [1]), FedRN (CIKM 2022 [2]), FedNoRo (IJCAI 2023 [3]).

[1] Xu, Jingyi, et al. "Fedcorr: Multi-stage federated learning for label noise correction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Kim, Sangmook, et al. "FedRN: Exploiting k-Reliable Neighbors Towards Robust Federated Learning." Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.

[3] Wu, Nannan, et al. "FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneity." arXiv preprint arXiv:2305.05230 (2023).

• Looking at the results in Table 3 and the limited effect of epsilon, it brings up the question: What is the actual effect of using the contrastive label
• Overall it is quite hard to follow the paper with the confusing notation:
  • usage of \hat, \tilde, \check, and \prime gets very unnecessarily confusing.
  • on page 4 authors write $-l(f(x_{n\prime}), \check{y}_n)$ , but in eqn 3 they suddenly switch to $-l(f(x_{n}), \check{y}_{n\prime})$
  • on page 5 section 4.2 what is $P(\tilde{Y}\vert\tilde{D})$
• I'm not quite sure what this means: "Intuitively, the “new” label has to be sampled globally; otherwise, the global information is missing and the negative effect of local openset label noise would induce performance degradation." There are no experiments to confirm or refute this claim. In recent papers [1] it's been observed that the addition of noise (e.g., data augmentation) can in fact improve results on heterogeneous data
• In section 5.1 authors mention that "mixup data augmentation methods introduce massive training cost". that does not seem to be the case as per the paper [2]. can the authors discuss this in more detail.

1. Azam, Sheikh Shams, et al. "Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASR." arXiv preprint arXiv:2309.13102 (2023).
2. Zhang, Hongyi, et al. "mixup: Beyond empirical risk minimization." arXiv preprint arXiv:1710.09412 (2017).

Some of the unclear descriptions in the paper are confusing. Although the author uses DP to share labels since the server knows T_DP and the server can obtain the user's label distribution p_c, the risk of privacy leakage still seems to exist.