Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages

We thank the reviewer for the insightful and positive feedback!

W1

the two tasks leveraged are primarily on high-resource languages, rather than low-resourced language. I would suggest to the authors to include more tasks - there are many low-resource language datasets (for instance on African languages MasakhaNEWS, Masakhaner (1.0 and 2.0 - which have been cited by the way but not used), MasakhaPOS; Indic languages: https://github.com/AI4Bharat/indicnlp_catalog; etc) and tasks.

Thank you for recommending these excellent datasets for our evaluation.

We agree that diversifying our dataset to include African and Indic languages will significantly strengthen our paper's scope and alignment with its title. To address this, we have initiated experiments with MasakhaNEWS and plan to conduct further research with MasakhaNER and IndicNLP Catalog datasets shortly.

MasakhaNEWS is a benchmark dataset for news topic classification covering 16 languages widely spoken in Africa, where African languages are severely under-represented in NLP research due to lack of datasets covering several NLP tasks. The task involves categorizing news articles into different categories like sports, business, entertainment, and politics. We choose English, Hausa, Kiswahili, French and Yorùbá in our preliminary experiments. We sample 1433 instances for training and 411 for evaluation sets for each language.

Table 2 in our revised paper presents the preliminary results of experiments focused on MasakhaNEWS. When employing Federated Prompt Tuning in comparison to the monolingual and centralized counterparts, in general, a significant gain in accuracy is observed when adopting our Federated approach compared with the monolingual baseline.

Q1

The Aggregation formula is a bit confusing. Did you mean h_{global, t+1} = \sum_{k=1}^{m} h_{k, t}? Because the t+1 on the last term does not make sense to me.

We appreciate your detailed feedback on the notation! We have revised the formula on page 5 and added more details to avoid any confusion in our updated version.

2. We emphasize the simplicity and effectiveness of our proposed paradigm, in which case modification or adjustment are not necessary but might make it complex.

In previous cases, data transmission was always one-directional. Existing approaches focus on solving this locally, for example, through cross-lingual transfer, as well as data augmentation and preference training to address these bottlenecks.

In our paper, we approach it from a collaborative (two-directional) perspective. By training LLMs collaboratively across multiple participants without sharing raw data, the accuracy, robustness, and generalizability of LLMs can be enhanced by leveraging collective knowledge and exposing models to a wider range of linguistic patterns.

There exists very little research from such a collaborative perspective for low-resource languages. Our findings open up new avenues for exploration and have the potential to inspire future research in this area.

Following the principle of Occam's razor, we adopt the concept of "federated" as a simple and established concept to describe our solution to the problem, which not only contributes a timely and practical solution to a rapidly evolving field but also vividly depicts the key innovation of our paradigm: knowledge sharing and aggregation without data transmission.

Additionally, from the federated learning perspective, as far as we know, we are the first paper to investigate the data efficiency and transferability brought by federated learning, and we believe this sheds some light on how federated learning can benefit LLM training in terms of generalizability and stability, beyond simply mitigating compliance risks.

[1] Joshi, Pratik, et al. The state and fate of linguistic diversity and inclusion in the NLP world. ACL 2020.

[2] Bérard et al., A Multilingual Neural Machine Translation Model for Biomedical Data NLP-COVID19 2020.

[3] Lauscher et al., From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers. EMNLP 2020.

[4] Xia et al., Generalized Data Augmentation for Low-Resource Translation. ACL 2019.

W3

There are better parameter-efficient finetuning methods, such as LORA/QLora, that the authors should conduct experiments on and do comparision with prompt tuning. The results show prompt tuning are much worse than full-federated tuning, thus casting doubt if the cost-saving is worth it.

Thank you for your valuable suggestions! Following the reviewer's constructive feedback, we have implemented experiments with LoRA (r=8, lora_alpha=16, lora_dropout=0.1) and summarized the results in the table below.

Table 4 in our revised paper presents the preliminary results of experiments on the NC task. Bold scores indicate the best performance between Prompt Tuning and LoRA in each column.

We also conducted a comparison of parameter efficiency and communication overhead in the NC task:

Additionally, we have included Figure 7 in Section 5.4 of our revised paper to clearly illustrate the comparison between Prompt Tuning and LoRA.

The results demonstrate that Federated LoRA and Federated Prompt Tuning achieve comparable performance, with Federated Prompt Tuning showing a slight advantage. In terms of data transmission and communication cost, Prompt Tuning requires only about 80% of the resources compared to LoRA, and leads to less privacy leakage.

Furthermore, as the pretrained model scales up, the performance of Prompt Tuning rapidly improves, approaching or even surpassing full finetuning. This indicates its significant potential. Prompt Tuning also enhances the overall model's generalization capabilities. The prompt encoder acts as a mechanism to extract linguistic-specific patterns on the client and general linguistic patterns on the server, showcasing advantages that adapters cannot match.

This justifies our use of Federated Prompt Tuning in our research, considering its efficiency in terms of parameters and communication, as well as its capability for generalization and adaptation to low-resource languages, which are crucial and undoubtedly worth it.

Regarding QLoRA, we did not consider quantization in our study since our focus is solely on updating the parameters of prompt encoders on clients and the server, keeping the pre-trained model frozen. QLoRA involves quantizing the pre-trained models, which falls outside the scope of our discussion and does not contribute to reducing communication costs, a key bottleneck in the federated setting.

W2

Though it may have implicitly inferred by the concept of FL, the paper did not mention why and how federated learning helps with privacy and in which case one should use FL for their application. The purpose of the task of multilingual finetuning in this case, is not warranted use case of privacy preservation. There is no reported evidence that privacy is actually preserved. Such as whether the final model memorize the local data.

We thank the reviewer for the insightful comments and concerns regarding privacy! We appreciate the opportunity to clarify this aspect of our work. It's important to note that multilingual finetuning here is not an approach for preserving privacy but rather a problem we aim to solve.

1. Our approach inherently supports data privacy, specifically by complying with international data privacy regulations. This compliance minimizes the need for cross-border data transmission, ensuring legal compliance and facilitating collaboration among entities with limited local computing resources, as detailed in Section 3.2.

2. We directly address privacy concerns by reducing the volume of transmitted data, thereby limiting potential privacy breaches. As demonstrated in Section 5.4, transmitting fewer parameters significantly reduces the risk of privacy leakage, aligning our methodology with the privacy focus highlighted in the abstract.

3. We would like to provide evidence for alleviating memorization from the following aspects:

(1) By freezing the core language model parameters, we prevent the model from altering its foundational understanding of language. Consequently, the prompt encoder reduces the risk of memorizing specific lexical cues and spurious correlations, as discussed in Section 5.1 [1].

(2) Components of Federated Learning play an essential role in reducing unintended memorization [2]. Specifically, clustering data according to users—a key design element in FL—significantly reduces such memorization. Additionally, using the Federated Averaging method for training further decreases the risk.

4. Regarding privacy protection, we acknowledge that we did not add extra privacy protection techniques to defend against potential privacy attacks, such as gradient inversion. Therefore, we appreciate the reviewer's valid points and have revised our paper to clarify our contribution to privacy and removed related claims about the capability of privacy protection to avoid confusion. However, various methods like secure aggregation (SA) and differential privacy (DP) can be applied in conjunction with our pipeline to further enhance privacy protection.

[1] Lester, Brian, Rami Al-Rfou, and Noah Constant. "The Power of Scale for Parameter-Efficient Prompt Tuning." EMNLP 2021.

[2] Thakkar et al. "Understanding Unintended Memorization in Language Models Under Federated Learning." PrivateNLP 2021.

please see our response to W5 in our first comment

W1

The proposed is a very trivial combination of federated learning and prompt tuning, which both are established methodology in their own realm. There is no novelty, such as modification or adjustment to the method that may have give a better results. In other words, people with an objective to do federated learning for privacy purpose can easily come up with prompt tuning as a solution to reduce costs.

We appreciate the opportunity to address the concerns raised by the reviewer and would like to defend our proposal, emphasizing its novelty and significance. In summary, we would like to clarify that our paper introduces federated prompt tuning as a solution to help address the linguistic and geographic boundaries hindering the application of LLMs to various regions and lower-resource languages.

We would like to further clarify from the following two aspects:

1. We emphasize the unique background and pressing need of our work, as we noticed the review may overlook the part of multilingual and low-resource language in the initial review.

First, our work primarily focuses on the under-representation of multilingual and low-resource languages in large language models. As natural language processing technologies advance, not all languages have been treated equally by developers and researchers. There are around 7,000 languages spoken in the world, and approximately 400 languages have more than 1 million speakers. However, there is scarce coverage of multilingual datasets. This is especially true for low-resource languages, where data scarcity is a major bottleneck. Furthermore, the under-indexing of certain languages is also driven by access to compute resources. Mobile data, compute, and other computational resources may often be expensive or unavailable in regions that are home to under-represented languages. Unless we address this disproportionate representation head-on, we risk perpetuating this divide and further widening the gap in language access to new technologies. One pressing example is biomedical data. Due to its global scale, this digital content is accessible in a variety of languages, yet most existing NLP tools remain English-centric. This situation highlights the need for effective strategies: how can we exploit abundant labeled data from resource-rich languages to make predictions in resource-lean languages?

Also, we wanted to highlight the urgency and timeliness of the problem. The problem is very timely compared to other application scenarios. It was not even considered a year ago. Previously, due to the smaller size of language models, the demand for data was not as high, and different kinds and sources of data were treated equally. Currently, the progress of LLMs, their usability, the amount of attention they receive, and the increased regulation on data, compound and lead to the urgency of this problem, where we are among the first batch to attempt to break both lingual and physical barriers.

Q1

Citation formet incorrect, \citep{} be used to produce something like (Abc, et al., 2023) and not Abc, et al., 2023 everywhere.

Thanks for pointing it out. We have corrected the citations for all references in our revised paper.

Q2

Many grammatical errors, such as "Throughout the fine-tuning...""

We appreciate your feedback on the grammatical errors. We have revised the grammar to avoid any confusion in our updated version.

W4

Other generative and knowledge-based tasks, such as QA, translations and summarizations should be performed.

We appreciate the feedback. Our current paradigm is general-purpose and can be easily adapted to other generative and knowledge-based tasks. In response, we have expanded our evaluations to encompass a broader range of scenarios, addressing the concern of limited task selection. This rebuttal is part of a series, and we will provide additional results during the discussion period.

• Low-resource Dataset

Our setting and results: MasakhaNEWS, covering 16 languages widely spoken in Africa, where African languages are severely under-represented in NLP research due to a lack of datasets covering several NLP tasks. The task involves categorizing news articles into different categories like sports, business, entertainment, and politics. We chose English, Hausa, Kiswahili, French and Yorùbá in our preliminary experiments. We sample 1433 instances for training and 411 for evaluation sets for each language.

Table 2 in our revised paper presents the preliminary results of experiments focused on MasakhaNEWS. When employing Federated Prompt Tuning in comparison to the monolingual and centralized counterparts, in general, a significant gain in accuracy is observed when adopting our Federated approach compared with the monolingual baseline.

• Question Answering

Dataset and our setting: MultiLingual Question Answering (MLQA) is a benchmark dataset for cross-lingual question-answering performance, covering English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. We sample 1433 instances for training and 411 for evaluation sets for each language.

• Machine Translation

Dataset and our setting: UN Corpus is a Machine Translation dataset of official records from the UN proceedings over the years 1990 to 2014, covering six languages: English, French, Spanish, Russian, Chinese, and Arabic. we sample 10k in each direction for training and 5k each for evaluation sets. We cover three machine translation directions: En → Fr, Ar → Es, Ru → Zh, and sample 10k in each direction for training and 5k each for evaluation sets.

Once we get the result, we’ll update our response and our paper.

We are pleased to present our preliminary results for Machine Translation. The process took more time than the classification tasks.

Our pre-trained model is M2M-100 with 418M parameters, a many-to-many MT model capable of translating between any pairing of 100 languages.

The table below displays our preliminary results on fine-tuning a machine translation model using the UN Corpus dataset. The scores are measured with sacreBLEU; higher scores indicate better performance.

We can use the pre-trained model as a baseline without fine-tuning. In all cases, fine-tuning demonstrates improvement compared to using the pre-trained model directly. Additionally, our federated prompt tuning paradigm outperforms its centralized counterpart (33.4 avg. BLEU for Ours vs. 32.5 for Centralized), demonstrating the superiority of our proposed paradigm.

Q

In section 5.4.1 In both the NC and XNLI tasks, despite the total number of parameters exceeding 278 million, the trainable parameters are only around 1.2 million, accounting for less than 0.5% of the total. Could the authors clarify which part of the model is being fine-tuned?

Yes, we clarify that we only update the prompt encoders. This includes the parameters of the local prompt encoders $h_k$ on Client k, and the parameters of the global encoder $h_g$ on the server in the revised paper (referred to as $h_{global}$ in the original paper). During this process, we keep the pre-trained language models frozen at all times.

Therefore, the trainable parameters are solely those within the prompt encoders, in contrast to the total number of parameters involved in full fine-tuning.

To further clarify and avoid any confusion, we have revised some details in Section 3. Additionally, we have attached a figure (Figure 2) in the revised version of the paper, illustrating the architecture of our prompt encoder and the tuning process.

W1

poor presentation: the citations are not separable enough from the main text, e.g., without any parenthesis, rendering the submission unreadable. Against the tradition and ease of reading, abbreviations are not defined in advance, e.g., NLI, PFL, PLM.

We apologize for any confusion caused by the current citation format. We have corrected the citations for all references in our revised paper.

We realize the oversight in not defining certain abbreviations, such as NLI (Natural Language Inference), PFL (Prompt Federated Learning), and PLM (Pre-trained Language Models), at their first occurrence in the text. We appreciate you highlighting this point. In the revised paper, we have ensured that all abbreviations are defined upon their first use.

W2

claims unverifiable: no code release.

We provide an anonymized version of the code repository, accessible through this link: https://anonymous.4open.science/r/Breaking_Physical_and_Linguistic_Borders-F1C5.

W3

conflating existing metrics with innovation: language distance is not a new concept.

Thank you for your insightful comments on our paper.

We acknowledge and agree with your review that the concept of language distance is not novel, having been explored in various contexts previously. However, we emphasize that our work introduces this concept within a unique and specific scenario: multilingual federated tuning. Our novel application provides a fresh perspective on language distance as a metric that illustrates the transferability relationships in multilingual NLP. This enables us to analyze the performance of federated prompt tuning and local monolingual transfer learning for low-resource languages.

To avoid overemphasizing the novelty of the language distance concept itself, we've amended the language in the abstract from "introduce language distance as a new concept" to "present a new notion of language distance" in our revised paper. This modification more accurately reflects our contribution.

We refer to [1] and [2] for our language distance measurement. Furthermore, we are open to learning about any additional references or works related to language distance that the reviewers may be aware of. If there are specific references that we have overlooked or that could further strengthen our work, we welcome their inclusion to enhance the comprehensiveness and depth of our research.

[1] Malaviya et al., Learning Language Representations for Typology Prediction. EMNLP 2017.

[2] Littell et al., URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. EACL 2017.

W4

conceptual weakness: the contrived baseline was bound to give the proposed approach an edge due to lack of federated learning. Also, what the paper refers to as prompts are just classifier model input, which are different from decoders-style LLM prompts as commonly acknowledged. Finally, the approach has absolutely nothing to do with privacy which the abstract and the main body consistently bolsters.

Regarding concerns about baselines

We appreciate the opportunity to clarify this aspect of our work.

In previous cases, data transmission was always one-directional. Existing approaches focus on solving this locally, for example, through local transfer with monolingual data.

In our paper, we approach it from a collaborative perspective, which we call federated prompt tuning in our paper. By training LLMs collaboratively across multiple participants without sharing raw data, the accuracy, robustness, and generalizability of LLMs can be enhanced by leveraging collective knowledge and exposing models to a wider range of linguistic patterns.

As you mentioned, there exists very little research from such a collaborative perspective for low-resource languages. Our findings open up new avenues for exploration and have the potential to inspire future research in this area.

What we would like to demonstrate is not simply the performance boost. It’s the data efficiency (Section 5.2) and transferability for different language similarities (Section 5.3) of our paradigm’s superiority on low-resource languages.

Regarding concerns about prompts

We would like to clarify that the prompts in our paper are NOT the same as classifier model input, and they are suited for all decoders-style LLMs. To further clarify the prompt tuning procedure and the prompt construction, we've added more details in Section 3 and Appendix B, and C in the revised version.

Instead of selecting discrete text prompts in a manual or automated fashion, in our paradigm, we utilize virtual prompt embeddings that can be optimized via gradient descent. Specifically, each prompt encoder, whether global or local, takes a series of virtual tokens, which are updated during tuning to better aid the model.

Figure 2 in our revised paper shows how our prompt tuning works on both clients and the server. Specifically, a textual prompt tailored for a specific task and input text is passed to the model. The task-specific virtual tokens are retrieved based on the textual prompt. With the input text tokenized, the discrete word token embeddings are retrieved. Then virtual token embeddings are inserted among discrete token embeddings and passed together into the pretrained models. Therefore, the prompt is adaptable across various pre-trained model architectures, including decoder-style, encoder-style, or encoder-decoder style.

Regarding concerns about privacy

We appreciate the opportunity to clarify this aspect of our work.

• Our approach inherently supports data privacy. Specifically, it complies with international data privacy regulations by minimizing the need for cross-border data transmission. This not only ensures legal compliance but also facilitates collaboration among entities with limited local computing resources, as detailed in section 3.2.

• We directly address privacy concerns by reducing the volume of transmitted data, thereby limiting potential privacy breaches. As demonstrated in section 5.4, transmitting fewer parameters significantly reduces the risk of privacy leakage, aligning our methodology with the privacy focus highlighted in the abstract.

We are pleased to present our preliminary results for Machine Translation. The finetuning took more time than the classification tasks.

Our pre-trained model is M2M-100 with 418M parameters, a many-to-many MT model capable of translating between any pairing of 100 languages.

The table below shows our preliminary results on fine-tuning a machine translation model using the UN Corpus dataset. The scores are measured with sacreBLEU; higher scores indicate better performance.

We can use the pre-trained model as a baseline without fine-tuning. In all cases, fine-tuning demonstrates improvement compared to using the pre-trained model directly. Additionally, our federated prompt tuning paradigm outperforms its centralized counterpart (33.4 avg. BLEU for Ours vs. 32.5 for Centralized), demonstrating the superiority of our proposed paradigm.

Thanks so much for your valuable comments and feedback.

Q1

Do you have any findings on why multilingual centralized learning is far worse than federated learning in Table 2?

Yes. This phenomenon has also been observed in previous works on Federated Learning [1]. Here are some possible reasons (Section 5.1, Page 7):

Firstly, Federated Learning has a weight averaging effect via the aggregation of clients’ models, which could increase the generalization of the global model, further leading to higher performance [2]. Additionally, by freezing the core language model parameters and only learning the prompt representations, prompt tuning reduces the model’s ability to overfit a dataset by memorizing specific lexical cues and spurious correlations [3].

These reasons demonstrate the superiority of our federated prompt tuning over the traditional centralized finetuning paradigm, offering a more robust and generalizable approach for low-resource languages.

[1] Rehman, Yasar Abbas Ur, et al. "Federated self-supervised learning for video understanding." ECCV 2022.

[2] Izmailov, Pavel, et al. "Averaging weights leads to wider optima and better generalization." UAI 2018 .

[3] Lester, Brian, Rami Al-Rfou, and Noah Constant. "The power of scale for parameter-efficient prompt tuning." EMNLP 2021 .

Q2 & W2:

Lack of clarity. The paper does not provide enough information about how the prompts are constructed or look like and hyperparameters for all settings. I suggest adding the information to the paper or appendix.

How did you tune the training and parameter averaging?

To further clarify the prompt tuning procedure and the hyperparameters, we've added more details in Seciton 2 and Appendix B, C in the revised version.

Virtual Prompt

We have also included a detailed figure 2 in our revised paper to more clearly show how the prompts are constructed and tuned.

In general, instead of selecting discrete text prompts in a manual or automated fashion, in our Multilingual Federated Prompt Tuning paradigm, we utilize virtual prompt embeddings that can be optimized via gradient descent. The primary objective of each **prompt encoder **is to generate an effective prompt embedding for each client based on task specific virtual tokens, to guide the PLM in producing the desired outputs.

Figure 2 in our revised paper shows the how our prompt tuning works on both clients and server. Specifically, a textual prompt tailored for a specific task and input text are passed to the model. Then task specific virtual tokens are retrieved based on the textual prompt. With the input text tokenized, the discrete word token embeddings are retrieved. Then virtual token embeddings are inserted among discrete token embeddings and passed together into the PLM.

Federated Prompt Averaging

In every communication round $t$, Federated Prompt Averaging includes the following steps.

Initialization: The server initializes the global prompt encoder $h_g^{t}$. Each client initializes its local prompt encoder $h_0^{t}, h_1^{t}, \ldots h_k^{t}$.

Client Selection: We select a fraction $C$ of the total $K$ clients for training. This subset size is $m = \max(C \times K, 1)$. The subset we choose is denoted as $S$.

Local Encoder Tuning: Each client $k$ in $S$ fetches the current global prompt encoder $h_g^{t}$, and assembles it with the PLM. During the local training on the local data $\mathcal{D}_k$, The PLM's parameters stay fixed while local prompt encoder parameters $h_k^{t}$ are tuned.

Aggregation: The server aggregates updates from all clients using weighted average. The global prompt encoder $h_g^{t+1}$ is updated based on the received parameters $h_k^{t}$ from clients for the next round of federated prompt tuning:

Hyper-parameters

For all of the experiments, we report results using the 1e-3 learning rate, and we use early stopping (5 epochs of no improvement).

For FL experiments, we adjust the parameter $\alpha$ that controls the mixture of languages in the dataset. An $\alpha$ value of 1.0 signifies a uniform mixture of all languages, while values closer to 0 indicate a dominant representation of individual languages or a more separated mixture.

When we use Prompt Tuning to optimize the parameter efficiency, the prompt tuning init text is Predict the category given the following news article for all the News Classification tasks. By providing the string of words, we initialize virtual token embeddings from existing embedding weights. This string is tokenized and tiled or truncated to match the number of virtual tokens, which is 1 in our experiments.

W1:

In terms of novelty, the proposed idea is not new, and it is only a further investigation of the multilingual setting.

We would like to kindly defend our proposal.

To further clarify the significance and originality of our work, we've added our motivation with Multilingual NLP Background in Appendix A and the Contribution paragraph in Section 1 in our updated paper.

In the paper, we introduced federated prompt tuning as a solution to help address the linguistic and geographic boundaries hindering the application of LLMs to various regions and lower-resource languages. Here we would like to provide some clarification about the motivation and significance of our research in the following two aspects.

Multilingual NLP and Low-resource Languages

As natural language processing technologies advance, not all languages have been treated equally by developers and researchers. There are around 7,000 languages spoken in the world, and approximately 400 languages have more than 1 million speakers. However, there is scarce coverage of multilingual datasets. This is especially true for low-resource languages, where data scarcity is a major bottleneck. Furthermore, the under-indexing of certain languages is also driven by access to compute resources. Mobile data, compute, and other computational resources may often be expensive or unavailable in regions that are home to under-represented languages. Unless we address this disproportionate representation head-on, we risk perpetuating this divide and further widening the gap in language access to new technologies [1].

One pressing example is biomedical data. Due to its global scale, this digital content is accessible in a variety of languages, yet most existing NLP tools remain English-centric [2].

This situation highlights the need for effective strategies: how can we exploit abundant labeled data from resource-rich languages to make predictions in resource-lean languages?

Timeliness of breaking physical and linguistic barriers in the LLM era

We wanted to highlight the urgency of the problem. The problem is very timely compared to other "application scenarios." It was not even considered a year ago. Previously, due to the smaller size of language models, the demand for data was not as high, and different kinds and sources of data were treated equally. Currently, the progress of LLMs, their usability, the amount of attention they receive, and the increased regulation on data, compound and lead to the urgency of this problem, where we are among the first batch to attempt to break both lingual and physical barriers.

In previous cases, data transmission was always one-directional. Existing approaches focus on solving this locally, for example, through cross-lingual transfer, as well as data augmentation and preference training to address these bottlenecks [3, 4].

In our paper, we approach it from a collaborative perspective. By training LLMs collaboratively across multiple participants without sharing raw data, the accuracy, robustness, and generalizability of LLMs can be enhanced by leveraging collective knowledge and exposing models to a wider range of linguistic patterns.

There exists very little research from such a collaborative perspective for low-resource languages. With data and computing power being very important yet limited for LLMs, we've never needed such a lightweight collaborative paradigm more urgently than we do right now.

So we introduce the concept of "federated" as a simple and established progression to our problem, which not only contributes a timely and practical solution to a rapidly evolving field, but also vividly depicts the key innovation of our paradigm: knowledge sharing and aggregation (double direction) without data transmission.

Additionally, from the federated learning perspective, as far as we know, we are the first paper to investigate the data efficiency and transferability brought by federated learning, and we believe this sheds some light on how federated learning can benefit LLM on training generalizability and stability, beyond simply mitigating compliance risks.

[1] Joshi, Pratik, et al. The state and fate of linguistic diversity and inclusion in the NLP world. ACL 2020.

[2] Bérard et al., A Multilingual Neural Machine Translation Model for Biomedical Data NLP-COVID19 2020.

[3] Lauscher et al., From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers, EMNLP 2020.

[4] Xia et al., Generalized Data Augmentation for Low-Resource Translation ACL 2019.

Suggestions:

Figure number is missing on Page 2 "As depicted in Figure , " Missing Figure/Table "This translates to over 99% reduction in the communication overhead shown in 3" Typo "Finetuning accuracy across different lanugages on the NC task."

We appreciate your detailed suggestions on the typos and missing figure/table numbers! We have fixed them all in our updated version.

Dear Reviewer LsRx,

We would like to sincerely thank you again for your time in reviewing our work!

We understand you might be quite busy. However, as the discussion deadline is approaching, would you mind checking our response and confirming whether you have any further concerns or questions? Any further comments and discussions are welcomed!

Best Regards,

The authors of Paper1909