Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning

Thank you very much for your appreciation and detailed comments! We make the following responses point by point to address your comments.

W1: "While the "Contrast" part of the proposed method is focused around preserving privacy, it would have been nice to have some verification of this, as privacy is absolutely central to the proposed model."
Q2: " Verification: in real life, you would never use ATC w/o "Contrast", right? Wouldn't this put customer data at risk? Regarding the point I made in the "Weaknesses" section, is contrastive learning alone enough to safe guard against privacy leakage?"

Thank you for the comments!

• The main purpose of the "Contrast" part is to group clients based on their local objectives rather than preserving privacy. It is used to encourage clients to exchange useful downstream task knowledge by federated aggregation.
• For privacy protection, the proposed ATC provides fundamental privacy protection (no sharing of data directly) with the help of the federated learning framework. Besides, privacy protection can be enhanced by privacy protection algorithms (such as differential privacy, homomorphic encryption, etc.) to satisfy flexible protection requirements from different applications. Users can balance utility-privacy-cost according. These privacy protection algorithms can be regarded as plugins and are orthogonal to the proposed framework, thus in this submission, we pay more attention to describing how can collaborate clients with different learning objectives.

We have made the above discussions more clear in the revised paper, thank you again!

W2: "In the Introduction, the paper introduces the need for improved federated learning by essentially complaining that increased data protections hurt "data owners". This doesn't sound great...."

We sincerely appreciate your valuable suggestions!

We absolutely agree that user private data should be protected and some actions should be taken to keep companies from using personal data in risky ways! Our submission explores a practical solution for training models in a privacy-preserving manner. We have modified the introduction according to your suggestions to make it precise and friendly. Thank you very much!

Q1: "Why do you not experiment with more clients? It's unclear to me if these are simulated "clients", or real-life clients."** Thanks for your comment!

• We adopt simulated clients in the experiments.
• We conduct an experiment with more clients: the number of clients increased fivefold in each dataset. The results demonstrate that the proposed method achieves noticeable improvement by 4.94% and 4.75% on the overall performance compared to PerCFL and SPFL (two of the strongest baseline methods), respectively. These results further confirm the effectiveness of the proposed method.

---

Thank you again for the detailed comments and valuable suggestions! We have made these modifications in the revised paper and believe this submission has been further improved according to your helpful suggestions, and hope that these responses can address all your concerns and convince you to lean more toward acceptance of the paper.

Thank you very much for your appreciation and detailed comments! We make the following responses point by point to address your comments.

W: "No statistical significance testing conducted for the results that appear quite incremental compared to the baselines."

Thank you for the suggestions! We have added the statistical significance testing in the revised paper in Section 4.4. Specifically, we conduct an independent t-test between the results of the proposed method and PerCFL (the strongest baseline) to show the significance and get $p< 0.01$.

Q1: "In the introduction, kindly provide a concrete example/use case for federated learning with heterogeneous tasks."

Thank you very much for the suggestion! Federated learning with heterogeneous tasks has many practical use cases. For example, multiple companies may collect text data from similar domains, such as news, but utilize these data for different downstream tasks, such as event discovery, sentiment analysis, and so on. With the help of federated learning and the proposed framework, these companies can collaboratively train a global model to learn valuable and generalized knowledge from text data.

We have added the example in the revised paper, thank you again!

Q2: "What does consistent model update mean as given in the abstract? "

Thanks for your comment! The term "consistent" means that the update directions are similar and not conflict.

Q3: "Section 3.1: Shed Lighted -> Based on such insights"

Thanks for your suggestion! We have modified it accordingly.

Q4: "Section 3.2 - Extension with Clustering Algorithms: The gradients change with each iteration, so can the clusters change as well, or is it fixed based on the first iteration? "

Thanks for your comment! The clusters might change as the gradients change with each iteration.

Q5: "Section 3.2 - Extension with Clustering Algorithms - Kindly extend upon the clustering algorithm - provide more details."

Thank you for the comments. The clustering algorithm used in this study is an agglomerative clustering algorithm proposed by [1], which serves as an example to demonstrate the extensibility of our framework. In fact, more advanced clustering algorithms can be adopted in the assign training stage. We kindly encourage readers to concentrate on the overall architecture of the framework and refer to the references for further details if necessary. Thank you again!

Q6: "Section 3.3 - 4th paragraph - “To further..” is confusing. Kindly provide concrete examples to highlight this important stage. "

Thank you for the comments! In section 3.3 - 4th paragraph, we introduce a synthetic data generation approach for contrastive learning. This approach is particularly valuable in scenarios where the proposed framework is applied to downstream tasks involving sensitive data (e.g., healthcare) or non-IID data. It is because finding an eligible public dataset can be intractable.

We have made them more clear accordingly in the revised paper, thanks again!

Q7: "Section 3.3 - Mention that the contrastive learning loss is one of the additional objective other than the client - task specific objective. "

Thank you for the suggestions! We have modified it accordingly in Section 3.3 for a better understanding of the contrastive learning loss.

Q8: "Section 4.2: In FEDAVG what equations govern the federated aggregation? In FEDPROX what is the proximal term "

Thank you for the comments!

• In FEDAVG, the equations of federated aggregation can be given as $\boldsymbol{w}^{(t+1)} \leftarrow \boldsymbol{w}^{(t)} + \sum _{i \in [N]} \frac{|D _i|}{\sum _{j\in [N]}|D _j|} \Delta \boldsymbol{w}^{(t)} _{i}$
• In FEDPROX, the proximal term can be given as $\frac{\mu}{2}||\boldsymbol{w} - \boldsymbol{w}^{(t)}||^2$ (please refer to the algorithm 2 in [2] for more details).

Q9: " Table 2: Are these numbers statistically significant? Kindly conduct a significance testing."

Thank you for the suggestions! We have added the statistical significance testing in the revised paper in Section 4.4.

Q10: "In the related work section, please highlight the importance of ATC for each of the different previous works rather than clubbing them in the end."

Thank you for the suggestions! We have modified the related work section accordingly for better highlight the importance of ATC.

Q11: "In the conclusion section, kindly mention quantitative results, instead of writing the abstract again."

Thank you for the suggestions! We have provided more quantitative results in the conclusion section.

---

Thank you again for the detailed comments and valuable suggestions! We have made these modifications in the revised paper and believe this submission has been further improved according to your helpful suggestions, and hope that these responses can address all your concerns and convince you to lean more toward acceptance of the paper.

Refs:
[1] Modern hierarchical, agglomerative clustering algorithms. 2011.
[2] Federated optimization in heterogeneous networks. MLSys, 2020.

Thank you very much for your appreciation and detailed comments! We make the following responses point by point to address your comments.

W1: "The contrast part of the framework lacked rigorous motivation. There was no discussion on the pros and cons of treating the cosine similarity of the model updates as a good measure of grouping the data domains. Are the model updates representative of not only the variety of the data domains but also the heterogeneity of the downstream tasks that are applied on that data? "

Thank you for the suggestions! The main purpose of the contrast part of the framework is to group clients based on their local objectives, encouraging them to exchange useful downstream task knowledge by federated aggregation. For more details:

• The contrast part is a critical step that builds upon the assign part and complements it. In the assign part, clients perform local training with the assigned self-supervised objectives, and their model updates are more related to their data domains; While in the contrast part, clients train with their own objectives, thus their model updates are more specific to their downstream tasks.
• We conduct experiments and have further discussions on how we group the clients in these two different training parts, as shown in Section 4.5. The results show that in the assign stage, clients with similar data domains are more likely to be aggregated together. On the other hand, in the contrast part, clients with similar learning objectives are grouped. These experimental results further confirm the main idea behind our design.

We have added the above discussions and made them more clear in the revised paper, thank you again!

W2: " Looking at table 2 showed the proposed framework in the top performing range among the competitors, but the improvement in many cases did not seem significant compared to the state of the art."

Thanks a lot for your comment. This study focuses on the collaboration of heterogeneous NLP Tasks, making it crucial for the proposed method to achieve better overall performance across multiple tasks, rather than just being competitive in one or a few tasks. As demonstrated in Table 2, the proposed method shows a significant improvement (around 1.5% compared to the strongest baseline) on the overall performance. For more details:

• While some baseline methods may achieve comparable performance in one or a few tasks, they fail in other tasks and consequently obtain lower overall scores, as shown in Table 2. For example, PerCFL (the strongest baseline) performs similarly to our proposed method on AGNews/NewsQA, but causes a significant performance drop of approximately 5%/2% on SQuAD/IMDB.
• In comparison to FedAvg-ST (which constructs multiple FL courses using FedAvg independently, with each FL course only involving clients with the same or similar tasks), the proposed method consistently achieves improvement. These results demonstrate the effectiveness of our method in collaborating heterogeneous NLP tasks.

Q: " It would be useful to explicitly mention the use of self-supervised learning objectives for the assign portion at the bottom of the introduction section ... Similarly, it would be useful to briefly mention the use of clustering approaches while contrasting."

Thanks a lot for your suggestions!

• The self-supervised learning objectives for the assign portion used in this study are Masked Language Modeling (MLM) and Denoising Reconstruction (DR), which have been widely adopted in the field of NLP (please refer to Section 3.2 at the bottom of page 3 for more details). Besides these two objectives, more advanced and general self-supervised learning objectives are also compatible with the proposed ATC framework.
• The clustering approaches while contrasting can be briefly described as: each client is aggregated with its K closet neighbors based on the gradient similarity. For more details, please refer to Section 3.3, located on page 5 near Equation (3) & (4).

We have added the above description according to your suggestions in the revised paper, thank you again!

---

Thank you again for the detailed comments! We believe this submission has been further improved according to your helpful suggestions, and hope that these responses can address all your concerns and convince you to lean more toward acceptance of the paper.

W5: "Although there is a marginal improvement it is stated "superior performance"

Thanks for your comments. The proposed method shows a significant improvement (around 1.5% compared to the strongest baseline) on the overall performance. We have made the description more clear in the revised paper.

W6: "Many writing issues"

Thank you very much for the suggestions regarding the paper writing. We have addressed these issues accordingly in the revised paper, and also made carefully polished to further improve the writing quality.

Q1: "How will this framework tackle backdoor attacks?"

Thanks a lot for your comments. The proposed framework can be enhanced by various approaches to tackle backdoor attacks, according to the user's requirements of balancing utility and privacy. On the one hand, inspired by previous study[3], we can utilize some simple yet effective approaches, such as norm clipping and differential privacy, to mitigate the backdoor attacks without hurting the overall performance. On the other hand, robust aggregation rules[4,5] can be applied to filter out malicious model updates.

We have added the above discussions in the revised paper, thank you again!

Q2: " Non-IID could have more discussion, especially the Non-IID introduced by multiple tasks and datasets."

Thank you very much for your suggestions! The Non-IIDness discussed in the context of FL can be defined as covariate shift and concept shift [6]: the marginal distributions $\Pr(X)$ or the conditional distribution $\Pr(X|Y)$ can vary across clients. The Non-IIDness becomes more complicated and mixed when it comes to heterogeneous tasks, where clients might have different yet related feature and label spaces.

---

Thank you again for the detailed comments! We believe this submission has been further improved according to your helpful suggestions, and hope that these responses can address all your concerns and convince you to lean more toward acceptance of the paper.

Refs:
[1] Federated Multi-Task Learning. NeurIPS, 2017.
[2] Personalized heterogeneous federated learning with gradient similarity. 2022.
[3] Can You Really Backdoor Federated Learning? NeurIPS 2019.
[4] Understanding distributed poisoning attack in federated learning. ICPADS, 2019.
[5] Machine learning with adversaries: Byzantine tolerant gradient descent. NeurIPS, 2017.
[6] FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. ICLR 2021.

Thank you very much for your appreciation and detailed comments! We make the following responses point by point to address your comments.

W1: "It is still constrained to the global tasks: depending on the server to assign a global task (meaning that participants need to abide by this global task or has to be intrinsically similar)"

Thank you for the comments! In this study, we have carefully selected global tasks (i.e., self-supervised language model pretraining tasks) with the intention of benefiting clients with heterogeneous NLP tasks, while avoiding any additional potential privacy concerns. These global tasks have been proven to be general enough when focusing on the field of NLP, and would not cause additional constraints.

W2: "Some novelty is overclaimed, e.g., FL + multi-task[1], there is no comparison with this baseline."

Thank you very much for the suggestion regarding the references.

• We notice that the mentioned paper primarily focuses on addressing issues of high communication cost, stragglers, and fault tolerance in general distributed multi-task learning. However, it may not be effectively applicable to collaboration heterogeneous NLP tasks without careful design, particularly in the case of NLU and NLG tasks. We have accordingly included the references and added the discussions, thanks again!
• We provide the discussions on the comparison between the suggested FL + multi-task framework and the proposed method in the related work section. Besides, in the experiments section, we provide empirical comparisons between the proposed method with the baseline[2] that adopts a multi-task framework, highlighting the advances of the proposed methods.

W3: "The SOTA of the experiments seems to be different than your experiments (ISOLATED). "

Thank you for the comments! We kindly point out that it is much more challenging under the context of federated learning than centralized training, since each client only owns a part of the entire dataset for training (i.e., ISOLATED). Besides, the adopted model is extremely small considering the communication cost in the simulation.

W4: "There is a concern about whether it is worth the margins of improvement provided by Contrastive Learning in the trade of privacy and communication cost"

Thank you for the comments!

• In the proposed ATC framework, the contrast training stage is a critical step that builds upon the assign training stage and complements it. In the assign training stage, clients perform local training with the assigned self-supervised objectives, and their model updates are more related to their data domains; While in the contrast part, clients train with their own objectives, thus their model updates are more specific to their downstream tasks. It is worth noting that different clients can benefit from the knowledge shared by others who have similar data domains or similar downstream tasks.
• The proposed ATC provides fundamental privacy protection (no sharing of data directly) with the help of the federated learning framework. Meanwhile, privacy protection can be enhanced by privacy protection algorithms (such as differential privacy, homomorphic encryption, etc.) to satisfy flexible protection requirements from different applications. Users can balance utility-privacy-cost according.
• The willingness of clients' participation is a really interesting and meaningful topic within the context of FL. We would like to take some exploration of incentive mechanisms in our future work. Thanks again for your valuable suggestions!

We have added the above discussions and made them more clear in the revised paper, thank you again!

To w3, you may still try your best effort to align the experiments with feasible baselines.

To w4, adding security protection will exacerbate the cost thus reducing the applicability of this proposal.

To w5, the baseline uses a reproduced score with is lower than the original paper. [Linked to w3]

To Q1, do you have some evidence to support the claim?

Thank you for your responses. I acknowledge they are taken into consideration.