Robust Training of Federated Models with Extremely Label Deficiency

We sincerely thank the reviewer for taking the time to review. We appreciate that you find that our paper has meaningful motivation, introduces a reasonable solution and the paper is well written. According to your valuable comments, we provide detailed feedback below and also add them to our revised paper. We hope that our response could address your concerns. We believe that these changes would improve the overall quality of our work.

Question 1: There are details missing in the method and experimental description, raising concerns about reproducibility. Regarding Equation (9), the authors did not explicitly specify the choices of the metric $d$ and the function $N$ in both the method and the experiments section. What's the specific selection of d and N in the experiments?

Answer 1: Thanks for pointing out this problem. According to your kind suggestion, we have added the following explanations to our revision.

• The function $N(\cdot)$ is used to quantify the relationship among samples. Specifically, $N(\cdot)$ is a matrix that calculates the distances among samples in the feature space, i.e., outputs of supervised model $Z_s \in \mathbb{R}^{n \times d}$ and those of unsupervised model $Z_u \in \mathbb{R}^{n \times d}$ with $d$ being the dimension and $n$ the batch size. Using these outputs, we can employ the matrix $M \in \mathbb{R}^{n\times n}$to represent the relationships among samples, i.e., $N(f(\mathbf{w}_s,\mathbf{x})):=M_s = Z_s \cdot Z_s^T, \quad N(f(\mathbf{w}_u,\mathbf{x})):=M_u = Z_u \cdot Z_u^T$.

• The metric $d(\cdot)$ is realized as the mean square error, which is formulated as: $d(N(f(\mathbf{w}_s,\mathbf{x})), N(f(\mathbf{w}_u,\mathbf{x}))) = \|| M_s - M_u\||^2$.

Question 2: The datasets are basic, and it appears that each experiment was run only once, with the random seed not being disclosed.

Answer 2: Thanks for your constructive comments. The random seed we use in our experiments is 0. Following your constructive question into account, we conducted the other two runs with different random seeds for both the baselines and twin-sight. The mean and deviation results are listed in the following:

We'll supplement more experiments and attach these results to our revision.

Question 3: The algorithm requires training an additional model, leading to increased computation and communication costs.

Answer 3: Thanks for indicating this shortage of twin-sight. Twin-sight exactly increases the communication and computation overheads. Following your valuable comments, we have added a detailed explanation of the extra overhead induced by our method. Specifically, our method doubles the training time and the communication cost, because we introduce an extra model.

Dear reviewer #37Jm,

Thank you for your valuable time reviewing our paper and providing insightful comments. We greatly appreciate your feedback, and in response to your comments, we provide thorough responses and add these valuable comments to our revision accordingly. We understand you are busy, so we have prepared a summary of our response for your convenience:

• (1) Detail of $N(\cdot)$ and metric $d(\cdot)$: Following your valuable comment, we add the explanations of the function $N(\cdot)$ and metric $d(\cdot)$. $N(\cdot)$ is a matrix that calculates the distances among samples in the feature space. The metric $d(\cdot)$ is realized as the mean square error.
• (2) Experimental issue: We have included the random seed used in our paper and have conducted two additional experiments with different random seeds. The mean and standard deviation of these experiments are provided in the table. In our revision, we will also include results from three other random seeds to enrich our results further.

We sincerely hope that our response has adequately addressed your concerns. We would greatly appreciate any further feedback from you. Your insights will be invaluable in helping us further enhance the quality of our work.

Best regards,

Authors of #5123

Dear Reviewer #37Jm,

Thanks very much for your time and valuable comments. We understand you're busy, and thus, we summarize our response.

We appreciate the opportunity to get your feedback, but the window for response is closing. Thus, we sincerely look forward to hearing from you, and we are very glad to provide answers following your comments.

Best regards and thanks,

Authors of #5123

We sincerely thank the reviewer for taking the time to review. We appreciate that you find that we consider a practically important problem, and propose a useful and principled solution and the paper is well written. According to your valuable comments, we provide detailed feedback below and also add them to our revised paper. We hope that our response could address your concerns. We believe that these changes would improve the overall quality of our work.

Question 1: Since "client drift" has been observed before, how "gradient cliff" is different from it, as the first contribution is claiming the phenomenon?

Answer 1: Thanks for pointing out this problem. According to your kind suggestion, we have added the following explanations to our revision.

• Client drift is caused by data heterogeneity while gradient conflict is typically induced by multiple objectives. In existing FSSL methods, the "client drift" and "gradient conflicts" often exist at the same time.
• Claiming the phenomenon is the motivation and the (first) contribution of this work since it motivates the proposed method. We would like to note that we did overlook the significance of this point. Thus, we sincerely thank you for your in-depth comments

Question 2: When exactly will "gradient cliff" happen? Does it happen all the time when training traditional FSSL methods? It would be nice to show how gradient similarities are improved by Twin-sight.

Answer 2: Thanks for pointing out the potentially confusing problem. Accordingly, we have added the following explanations to our revised paper.

• Gradient conflict is defined as the inconsistency between gradients, i.e., have a negative cosine similarity. The inconsistency may result from multi-objectives, e.g., multi-task learning [R1, R2, R3].
• In the context of label deficiency, gradient conflict may be attributed to both the samples and objectives. Specifically, partial samples have annotations and the objective on lableled and unlableled data are typically different. Advanced methods in FSSL typically use two different objective functions to train the local classifier, one for fully-labeled clients with cross-entropy loss and another for fully-unlabeled clients by unsupervised method. In this context, these methods suffer from gradient conflicts naturally.

In our experiments, gradient conflict often happens during the training process of traditional FSSL methods because of inconsistent loss function among different clients, please see Figure 2-(a). In contrast, we introduce a dual-model paradigm, avoiding the gradient conflict issue.

Reference:
[R1] Yu et al. Gradient surgery for multi-task learning. In NeurIPS, 2020.
[R2] Liu et al. Conflict-averse gradient descent for multi-task learning. In NeurIPS, 2021.
[R3] Wang and Tsvetkov. Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models. In ICLR, 2021

Question 3: It is nice to consider the partially labeled scenario, which is more realistic. The proposed method can be straightforwardly applied to a more realistic scenario where each client has a different ratio of labeled data. Are there any considerable difficulties? This is only for discussion, authors need not append experiments.

Answer 3: Thanks for your insightful comments.

• Our experiments show that our method can be effortlessly applied to partially labelled scenarios. Results shown in Table 3 demonstrate the effectiveness of our method.
• The mentioned more realistic scenario seems a promising direction for FSSL, where each client has a different ratio of labeled data. This could be regarded as the combination of partially labeled data and partially labeled clients. In this context, twin-sight can be applied in this scenario. Unfortunately, we failed to figure out such a practical scenario. We would like to explore this setting as our further work. Thanks for your enlightening advice again.

Question 4: What is the neighbourhood function used? Does it consider some graph structure?

Answer 4: Thanks for your valuable and insightful comments. We merely consider the distance among samples (in a batch) in the feature space. We believe employing a global relation among samples using the mentioned graph structure is a promising direction. We would like to explore it in our future work. Thanks for your kind suggestion.

Question 3: In the scenario of both unlabeled and labeled data, some combinations of federated algorithms and semi-supervised algorithms do not perform as well as federated learning using only labeled data, which violates normal cognition. The authors can try to explain the reasons for this phenomenon.

Answer 3: Thanks for your insightful comments. To our knowledge, some FSSL methods may lose their effect or even degenerate in a severe data heterogeneity environment. A recent work [R4] also indicates that we cannot guarantee an increase in the quantity of accurately labeled pseudo-labels during training in FSSL. For example, the pseudo-labeling technology sometimes imports many noise labels at some data distribution or dataset. These noises deteriorate the performance of the global model, leading to worse performance compared to not using the unlabeled data.

Question 4: The authors can try to prove the global convergence of the proposed objective function.

Answer 4: Thanks for your kind suggestion. We will leave it as our future work.

Reference:
[R1] Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[R2] Grill J B, Strub F, Altché F, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020.
[R3] Chen X, He K. Exploring simple siamese representation learning. In CVPR, 2021.
[R4] Diao E, Ding J, Tarokh V. SemiFL: Semi-supervised federated learning for unlabeled clients with alternate training. In NeurIPS, 2022.

We sincerely thank the reviewer for taking the time to review. According to your valuable comments, we provide detailed feedback below and also add them to our revised paper. We hope that our response could address your concerns. We believe that these changes would improve the overall quality of our work.

Question 1: This paper combines existing techniques to solve the federal semi-supervision problem, but lacks the necessary references and explanations. For example, the unsupervised algorithm adopted in formula (6) is not explained. $f(w_u;\cdot)$ as an unsupervised model, what are its specific working principles and objectives? What does $sim(\cdot)$ mean? What kind of neighbourhood relation construction function is $N(\cdot)$ of formula (9), and how does it construct neighborhood relation? For the parts that refer to other methods (in formula (6) and formula (9)), the authors should indicate the citation and briefly describe how them work.

Answer 1: Thanks for your detailed comments and we apologize for the missing references and detailed explanations. Accordingly, we have added the following explanations to the revised paper.

• For the unsupervised methods in Twin-sight, the $f(\mathbf{w}_u;\cdot)$ refers to the unsupervised model parameterized by $\mathbf{w}_u$. We performed some prevailing self-supervised methods, like SimCLR[R1], BYOL[R2], and SimSiam[R3] and SimCLR is choosen to be the backbone of $f(\mathbf{w}_u;\cdot)$. The results of different self-supervised methods are reported in Appendi B.2 How to determine the self-supervised method. And the $sim(f(\mathbf{w}_u;\mathbf{x}_i),f(\mathbf{w}_u;\mathbf{x}_j))$ denote the dot product between $\ell_2$ normalized embedding of data $i$ and $j$.

• The function $N(\cdot)$ is leveraged to quantify the relationship between different samples in a mini-batch. Specifically, $N(\cdot)$ is the matrix calculating the distance among samples in the feature space, i.e., outputs of supervised model $Z_s \in \mathbb{R}^{n \times d}$ and those of unsupervised model $Z_u \in \mathbb{R}^{n \times d}$ with $d$ being the dimension and $n$ the batch size. Using these outputs, we can employ the matrix $M \in \mathbb{R}^{n\times n}$to represents the relationships among samples, i.e., $N(f(\mathbf{w}_s,\mathbf{x})):=M_s = Z_s \cdot Z_s^T, \quad N(f(\mathbf{w}_u,\mathbf{x})):=M_u = Z_u \cdot Z_u^T$.

• The metric $d(\cdot)$ is realized as the mean square error, which is formulated as: $d(N(f(\mathbf{w}_s,\mathbf{x})), N(f(\mathbf{w}_u,\mathbf{x}))) = \|| M_s - M_u\||^2$.

We appreciate your valuable feedback. In the updated version, we will make sure to include references and provide clear explanations to enhance the overall readability and comprehensibility.

Question 2: Lack the necessary convincing. It is mentioned in the experimental setup that the paper will report the results after 500 rounds of training of the global model. What is the meaning of Round in Tables 1 and 2? If Round represents the number of training rounds, then why is the Round of each comparison method different and none reaches 500, which may cause the baseline results not being comparable. The authors can rereport experimental results and compare baseline performance with the same number of training rounds.

Answer 2: Thanks for your pointing out the confusing description. We have added the following explanation and fixed the problem in our revision.

• In our experiments, we use $R$ to denote the total communication rounds, which is 500 in the setup part. This means we train all the methods in 500 rounds for a fair comparison.
• In Tables 1 and 2, the 'Round' refers to the communication round required to reach the target accuracy. This can be seen in the footnote of Table 1. We use this metric to record the convergence rate of different methods, which means how many rounds the method can reach a target accuracy and 'None' denotes never reaching the target accuracy during $R=500$ rounds' training process.

Thanks again for your careful review.

Dear reviewer #NJbh,

Thank you for your valuable time reviewing our paper and providing insightful comments. We greatly appreciate your feedback, and in response to your comments, we provide thorough responses and add these valuable comments to our revision accordingly. We understand you are busy, so we have prepared a summary of our response for your convenience:

• (1) Detailed explanations issues: Following your valuable suggestions, we attach more references on the unsupervised methods. At the same time, we add detailed explanations about equations (6) and (9), the unsupervised methods, and the neighborhood function. We use the SimCLR method in our paper and calculate the distance among samples in the feature space to present $N(\cdot)$.
• (2) Experimental issue: We apologize for the confusion caused by the metric 'Round.' In the response, we provide a more detailed description of "Round" and all our results exactly experiment under the same "R" for a fair comparison.
• (3) Anti-consensus issue: We give the explanations that some other inappropriate approaches may instead introduce more noise labels, leading to worse results than if the unlabeled data were not utilized.

We sincerely hope that our response has adequately addressed your concerns. We would greatly appreciate any further feedback from you. Your insights will be invaluable in helping us further enhance the quality of our work.

Best regards,

Authors of #5123

We sincerely thank the reviewer for taking the time to review. We appreciate that you find that we propose a novel approach and the paper is well written. According to your valuable comments, we provide detailed feedback below and also add them to our revised paper. We hope that our response could address your concerns. We believe that these changes would improve the overall quality of our work.

Question 1: What are gradient conflicts? In vanilla supervised training gradients are averaged over all data. Certainly, there are some data points whose gradients differ in their direction. Is this a significant cause for concern? Has it been studied elsewhere?

Answer 1: Thanks for pointing out the potentially confusing concept. Accordingly, we have added the following explanations to our revision.

• Gradient conflict is defined as the inconsistency between gradients. The inconsistency may result from multi-objectives, e.g., multi-task learning (MTL) [R1, R2, R3].
• In the context of label deficiency, gradient conflict can be attributed to both the samples and objectives. Specifically, this occurs when only partial samples have annotations, and the objectives for labeled and unlabeled data differ. Advanced methods in FSSL typically use two different objective functions to train the local classifier, one for fully-labeled clients with cross-entropy loss and another for fully-unlabeled clients by unsupervised method. In this context, these methods suffer from gradient conflicts naturally.
• According to the definition of gradient conflict, two gradients are considered conflicting if they point away from one another, i.e., have a negative cosine similarity. This phenomenon also can be observed in Figure 2-(a) in our paper.

We appreciate your insightful question about the gradient conflict while answering the general question is challenging. We will try to list some related works.

• Gradient conflict caused by multi-objectives [R1].
• Gradient conflict caused by complex model architectures [R4].

The studies of gradient conflict can be found in the field of multi-task learning. The objective of MTL is the average loss of all kinds of tasks which typically leads to gradient conflict. The gradient conflict phenomenon has some drawbacks: i) this different loss may have various scales of gradient, and the largest one dominates the update direction; ii) The averaged objective can be quite unfavourable to a specific task’s performance [R2].

Reference:
[R1] Yu et al. Gradient surgery for multi-task learning. In NeurIPS, 2020.
[R2] Liu et al. Conflict-averse gradient descent for multi-task learning. In NeurIPS, 2021.
[R3] Wang and Tsvetkov. Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models. In ICLR, 2021
[R4] Li et al. Eliminating Gradient Conflict in Reference-based Line-Art Colorization. In ECCV, 2022.

Question 2: How is $N(\cdot)$ computed? What are the consequences of different choices of quantifying the neighbourhood?

Answer 2: Thanks for pointing out the problem. Accordingly, we have added the following explanations to our revision.

• $N(\cdot)$ is leveraged to quantify the relationship between different samples in a mini-batch.
• $N(\cdot)$ is the matrix calculating the distance among samples in the feature space, i.e., outputs of supervised model $Z_s \in \mathbb{R}^{n \times d}$ and those of unsupervised model $Z_u \in \mathbb{R}^{n \times d}$ with $d$ being the dimension and $n$ the batch size. Using these outputs, we can employ the matrix $M \in \mathbb{R}^{n\times n}$to represents the relationships among samples, i.e., $N(f(\mathbf{w}_s,\mathbf{x})):=M_s = Z_s \cdot Z_s^T, \quad N(f(\mathbf{w}_u,\mathbf{x})):=M_u = Z_u \cdot Z_u^T$.

Question 3: Can the authors shed some insight into why their method appears so much better than the baselines? "This constraint enables ... to collaborate and mutually benefit from each other’s strengths" is not a strong explanation. Why should the proposed coupling be better than manifold regularization or other forms of semi-supervised learning?

Answer 3: Thanks for your insightful comments. We would like to provide three key factors contributing to the improvement of our methods.

• We decouple the learning objective into two models which avoids gradient conflicts.
• In twin-sight interaction, our unsupervised model conducts an instance classification task which is a fine-grained classification problem. Namely, this task would contribute to the downstream classification tasks [R1]. Moreover, the data, model, and the objective function are consistent among all clients.
• Our supervised model conducts a classification task. Furthermore, the data, model, and objective functions are consistent among all clients, except for some unlabelled data paired with pseudo labels.

Reference:
[R1] Mitrovic et al. Representation Learning via Invariant Causal Mechanisms. In ICLR, 2021.

Question 4: Can you please include the default accuracy (accuracy achieved by always predicting the majority class).

Answer 4: Thanks for your detailed questions. The metrics we reported are under the global test dataset on the server side with a globally aggregated model. The global test dataset consists of 1000 data points at each class which means a balanced evaluation dataset.

Dear reviewer #r5bY,

Thank you for your valuable time reviewing our paper and providing insightful comments. We greatly appreciate your feedback, and in response to your comments, we provide thorough responses and add these valuable comments to our revision accordingly. We understand you are busy, so we have prepared a summary of our response for your convenience:

• (1) Comprehensive explanation: Following your constructive comments, we provide more details about gradient conflicts, which are most studied in multi-task learning.
• (2) Technical detail: For reproducibility, we give the calculation of $N(\cdot)$. On the other hand, we explain why we don't use the metric of default accuracy to compare the results.
• (3) Insightful issue: We give three key factors contributing to improving twin-sight. a) Twin-sight solves the gradient conflict naturally. b)unsupervised model conducts an instance classification task, a fine-grained classification problem. Furthermore, the data, model, and the objective function are consistent among all clients. c) supervised model conducts a classification task. Furthermore, the data, model, and objective functions are consistent among all clients, except for some unlabelled data paired with pseudo labels.

We sincerely hope that our response has adequately addressed your concerns. We would greatly appreciate any further feedback from you. Your insights will be invaluable in helping us further enhance the quality of our work.

Best regards,

Authors of #5123

Dear Reviewer #r5bY,

We sincerely thank you for your time and valuable comments.

We understand you're busy, and thus, we summarize our response. We appreciate the opportunity to get your feedback, but the window for response is closing. Thus, we sincerely look forward to hearing from you, and we are very glad to provide answers following your comments.

Best regards and thanks,

Authors of #5123