FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss

We sincerely thank the reviewer for their helpful comments and remarks. We have now addressed all of the concerns below.

• Concern about the communication load: The communication load is minor (please refer to A.3, Table 2 in our revised manuscript).
• Concern about too many hyper-parameters: The hyper-parameters of beta, augmentation method, and size of the mixed dataset remain the same as the existing papers (such as FixMatch, MixMatch, and SemiFL). As for the threshold value, it could be dataset-dependent. We selected the value 0.6 on CIFAR 10 and kept the same for other datasets to show the generalization of our method.
• Concern about the supervised baseline: For a fair comparison, we used a standard supervised baseline, where we did not conduct strong augmentation and the model was not pre-trained with anchor data (as mentioned in the caption of Table 1). Implementing a strong augmentation method is not so common in the benchmark results (such as in FedAdam [1], Fjord [2], and ZeroFL [3]), but for semi-supervised learning, people normally use a strong augmentation method, which we do the same.
• Description for Figure 3: Please see more explanations in the revised version of the paper, which we highlighted in Section 4.3.
• Concern about the absence of related work: Thanks for pointing out the papers. We have added FedCon as a baseline (please kindly refer to the general response point number 1 to all reviewers for more details) and included others in the background in our revised manuscript.
• Regarding other comments: Thanks for the feedback. We have updated our revised manuscript.

[1] Reddi, Sashank, et al. "Adaptive federated optimization." arXiv preprint arXiv:2003.00295 (2020).

[2] Horvath, Samuel, et al. "Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout." Advances in Neural Information Processing Systems 34 (2021): 12876-12889.

[3] Qiu, Xinchi, et al. "ZeroFL: Efficient on-device training for federated learning with local sparsity." arXiv preprint arXiv:2208.02507 (2022).

Dear Reviewer #6dEL,

Thanks very much for your time and valuable comments. We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirming whether you have any further questions? We are happy to provide answers and revisions to your additional questions. If our reply resolves your concern, please consider raising the score accordingly. Many thanks!

Best regards and thanks,

Authors

We sincerely thank the reviewer for their helpful comments and remarks. We now addressed all of the concerns below.

• Concern about novelty: We acknowledge that the general idea of label contrastive loss is not new, but we are the first to leverage it in federated semi-supervised learning to improve the quality of pseudo-labeling. Besides, all existing papers in federated/centralized SSL use the model-predicted results to generate pseudo labels. In our method, we for-the-first-time utilize the latent representation’s cosine similarity for pseudo-labeling and achieve a new SOTA performance. This provides a new solution for the general semi-supervised learning area.
• Concern about missing literature: Thanks for pointing out the papers. For [2], it is an earlier paper than SemiFL and performs worse than SemiFL, as we mentioned in the background section of our paper. Additionally, we have added FedCon as a baseline in our revised manuscript and included pFedKnow, FedIL, and FedSiam in the background section.
• Concern about the supervised baseline: For a fair comparison, we used a standard supervised baseline, where we did not conduct strong augmentation and the model was not pre-trained with anchor data (as mentioned in the caption of Table 1). This is the main reason supervised baseline performs poorly in some cases, especially when the anchor set is bigger.

Dear Reviewer #LCsM,

Thanks very much for your time and valuable comments. We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirming whether you have any further questions? We are happy to provide answers and revisions to your additional questions. If our reply resolves your concern, please consider raising the score accordingly. Many thanks!

Best regards and thanks,

Authors

We sincerely thank the reviewer for their helpful comments and remarks. We now addressed all of the concerns below.

• Concerns about other baseline comparisons. Please kindly refer to the general response to all reviewers.
• Concerns about the privacy of downloading the latent representations of the anchor data. In our paper, we assume that a small amount of labeled data will be stored on the server, which we call anchor data. We think it is a realistic and practical setup and is mentioned in the introduction. Even if we don’t assume the anchor data is public, since the size of the anchor embedding is significantly smaller than the raw data (we chose 128 in our experiments), it alone is not sufficient to reverse-engineer and reconstruct the raw data.
• Concerns about scalability. We have addressed this in the general response to all reviewers. Please kindly refer to point number 3 there.
• Concern about the use of Mixup. Mixup is commonly used in centralized semi-supervised learning settings and is a SemiFL component. We have shown the effect of mixup in our original manuscript (Table 1, Figure 2, 4, 5), named FedAnchor and FedAnchor (mixup).
• Regarding the font sizes, we will address this for the camera-ready version.
• Response to Equation 9. Our original equation is correct. Since $x_{mix}$ is already a mixup of $x^{fix}$ and $x^{mix}$, the $y$, as you mentioned, should be as “(x_{mix}, y^{fix})” for the first part.

Dear Reviewer #wVdB,

Thanks very much for your time and valuable comments. We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirming whether you have any further questions? We are happy to provide answers and revisions to your additional questions. If our reply resolves your concern, please consider raising the score accordingly. Many thanks!

Best regards and thanks,

Authors

We sincerely thank the reviewer for their helpful comments and remarks. We now addressed all of the concerns below.

• Concern about the novelty: We acknowledge that the general idea of latent representation and cosine similarity is not new in machine learning, which has been used in different areas and applications. The references provided by the reviewer mainly focus on dealing with data/label noise, which is not the concern of our paper. We for-the-first-time leverage the idea of latent representation’s cosine similarity to generate pseudo-labels in semi-supervised learning under the FL setup and achieve a new SOTA performance. The provided references can be good motivation papers, which we have cited in the background section (please see our new manuscript). Also, the method FedCon borrows the idea of using 2 models from BYOL without implementing any contrastive loss, despite the name.
• Contributions in addition to using FixMatch and MixMatch: 1. A unique pseudo-labeling method in FSSL described above; 2. A novel label contrastive loss to further improve the quality of pseudo labels.
• FedCon baseline: We have added this comparison in Table 1, section 4.2. Please kindly refer to general response point number (1) for this point.

We sincerely thank the reviewer for taking the time to review.

• Regarding the first question: If we look at the training curve, we can see that, for this case, the IID curve and the non-IID curve are very similar. We have also attached the plot showing the training curve to Appendix 4 in the newly uploaded manuscript. The testing accuracy between IID and non-IID is similar because, during each round, it also trains on the labeled and balanced anchor data in a supervised learning way. The training using anchor data is supervized and centralized. Also, we observe that solely training on the anchor data with strong augmentation techniques can achieve 40% for anchor size 5000. Through the observation, we can see that the FedCon baseline extracts limited information from the unlabeled data on the client side. Therefore, this reduces the effect of client data distribution (IID/non-IID) for FedCon.
• Regarding the second question: Since we would like to show that the choice of model architecture would not impact our method, we adopt the more popular and standard model architecture ResNet-18 in our manuscript, but in SemiFL, they use the Wide-ResNet model architecture. Resnet-18 is capable of the image classification task and identifies the "embedding" part of its architecture. Apart from the model architecture, the semiFL implementation, we follows their experimental settings.

We understand you're busy. But as the window for responding and paper revision is closing, would you mind checking our response and confirming whether you have any further questions? We are happy to provide answers and revisions to your additional questions. If our reply resolves your concern, please kindly consider raising the score accordingly. Many thanks!