LegoMT2: Non-Blocking Federated Learning for Massive Multilingual Machine Translation

Thanks for the valuable comments and positive feedback on the novelty and effectiveness of our work. Please find our point-to-point responses below, which we hope will address any questions or concerns you may have.

Question 1: What is the key difference between the federated learning used in this paper compared to the traditional federated method?

The key difference between the traditional federated method and our method is stated in Appendix A. The main difference includes three aspects: the data construction, the model architecture, and the communication method.

One key factor in recent advanced FL algorithms is communication compression [2,3,4,5,6,7]. The traditional implementation of FL requires that each client send a full model (or a full model update) back to the server in each round. For large models, this step is likely to be the bottleneck of FL for a few reasons.

Reason 1: internet speeds are usually faster for downloading than uploading. For example, Xfinity offers 125Mbps for downloads and only 15Mbps for uploads.

Reason 2: The size of the model is quite large. Therefore, there are numerous strategies available to compress these models or to reduce the bandwidth required for downloading the current model.

Reason 3: Different clients may experience a considerable amount of idle time while waiting for the server to collect all models and generate a new global model.

[1] Training mixed domain translation models via federated learning [2] Federated learning with compression: Unified analysis and sharp guarantees. [3] Federated learning: Strategies for improving communication efficiency [4] Communication-efficient adaptive federated learning [5] AutoFedNLP: An efficient FedNLP framework [6] FS-Real: Towards Real-World Cross-Device Federated Learning [7] FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction

Question 2-1: The final model used in inference is the averaged version of the 8 local models.

There seems to be a misunderstanding here: during inference (the results obtained in Table 1), we utilize only a single flow (mix-flow, 1.6B), not the averaged version of the 8 local models.

Question 2-2: The model size of the model should be viewed as 10.4B rather than 1.6B. Therefore, the model should be compared to the same-size finetuned model. Yes, we agree with your point. A fair comparing base model would be training 10.4B model and distilling into 1.6B model.

1) another valid comparison

We’ve given careful consideration to your suggestion and believe that, in addition to the model of the same size already listed in the table, there is indeed another model that could serve as a valid comparison:

• LegoMT2: In training, the model of the whole system is 10.4B; but in inference, each flow can work independently, and the model size is only 1.6B.

• Valid-Comparion: We train a 10.4B model with centralized method, and then distilled this model to 1.6B.

2) It can be anticipated that this method will lag behind LegoMT2 in terms of both speed and performance.

• Time: As evident from Table 3, training a 10.4B model is significantly slower, specifically 16.2 times slower, than LegoMT2. Furthermore, this pipeline also necessitates the distillation of the 10.4B model to a 1.6B model. This process requires the inference of the 10.4B model to obtain the output of each sample, which significantly increases the time requirement.

• Performance: The 54.5B model’s average performance surpasses LegoMT-2 by one point. Therefore, the upper limit of the 10.4B model has been established, and it’s important to note that the distillation process inevitably leads to performance losses.

Hence, when compared to the solution of training a 10.4B model and then distilling it, LegoMT2 proves to be both faster and more efficient.

The fair comparison is very important for a scientific research paper. We cannot just see the final report score, and more importantly, we need to care about why the improvement is achieved.

1. We didn't merely report the final score, but also carried out extensive analysis experiments focusing on the key design elements of this training framework.

• Table 4: model architecture analysis
• Table 5: the language group strategy analysis
• Figure 2: the effect of asynchronous learning algorithm on system performance.
• Figure 3: the effect of hyperparameters in the training algorithm

2. Extensive experiments, coupled with multi-faced evaluations, have underscored the efficacy of the framework.

• effectively supports translation between over 400 languages
• evaluation of existing benchmark (Table 1 & Table 2)
• human evaluation (Page 7-page 8 & Appendix D)
• Compare with ChatGPT (Page 20)

3. This task involves pre-training, which inherently requires substantial computational and time resources.

• It’s not that we are unwilling to make a fair comparison, but rather, we are unable to do so due to these constraints. | | Lego-MT2 | Valid-Comparison | | --- | --- | --- | | Training Speed | 16.2$\times$ | 1.0$\times$ | | Training Time | 15 days on 64 A100 GPUs | 240 days on 64 A100 GPUs | | distillation | - | Inference 10.4 to get the output of each sample | | Overall Time | 15 days | 240+ days |

• We can expect the performance of the solution (training a 10.4B model and then distilling to 1.6B) to be worse than LegoMT2.

• The upper limit of the 10.4B model has been established. The 54.5B model’s average performance surpasses LegoMT-2 by one point. Therefore

• It’s important to note that the distillation process inevitably leads to performance losses.

Hence, when compared to the solution of training a 10.4B model and then distilling it, LegoMT2 proves to be both faster and more efficient.

My question "why not train from scratch" shows my concern that your method may rely on a pre-trained NMT model.

1. Utilizing NLLB-200-1.3B as an initial step is important, but it is not enough for handling 400+ languages.

• Fine-tuning NLLB-200-1.3B directly on the data of over 400 languages actually results in a decrease in model performance, as shown in Table 1 of the paper.
• One of our contributions to utilizing NLLB includes effort in extending the vocabulary from 200 languages to 435 languages. The tokens for an additional 235 languages are initiated randomly, resulting 0.3B extra parameters compared to NLLB. In this way, we can best utilize the pre-trained parameters in NLLB while largely expanding the language support in LegoMT2, in a cost-effective way.

2. The success of NLLB-200-1.3B is not trivial. As shown below (which is from the NLLB paper[5]), they devote considerable energy to optimizing language performance. On average, enhancing the performance of a language takes 119 days, with the most time-consuming language requiring as long as 287 days for optimization. | | | | --- | --- | | # of Languages requiring Re-translation | 10 | | Avg # of Re-translations | 1 | | Max # of Re-translations | 2 | | Avg # of Days to Translate 1 language | 42 | | Avg # of Days to align | 28 | | Avg # of Days for 1 language | 119 | | Shortest Turnaround (days) for 1 language | 70 | | Longest Turnaround (days) for 1 language | 287 | | | |

[5] No Language Left Behind: Scaling Human-Centered Machine Translation

3. It is too costly to try full training of LegoMT2 without NLLB initialization.

• Plan 1: Spending 287 days to replicate the performance of NLLB, and then adding more than 200+ languages does not seem to be an efficient solution.
• Plan 2: Training each language from scratch and optimizing it, which takes over 200 days, is also costly.

Considering limited computation resources, we want to utilize them to conduct more impactful research.

Thanks for your helpful suggestions. We will make the necessary changes.

Weakness: Extensive experimental results and analyses are not fit in 9 pages. There are some description overlaps in Section 1 and 3 so the authors can move the contents from Appendix to the main pages.

Thanks for your suggestion! We will move the analysis of the difference between of traditional Federated Learning and comparison with LLM from the Appendix to the main part.

Question 1: Reg Section 3.3; how helpful is the parameter initialization with NLLB-200-1.3B? Have you ever looked into this effect, without having the NLLB initialization?

We have a reasonable belief that NLLB initialization is important. However, it is too costly to try full training of LegoMT2 without NLLB initialization. Therefore, we did not conduct alternative experiments. According to NLLB paper[1] (Table 2), it would take over 200 days to replicate the alignment process of NLLB to extend from NLLB's 200 languages to LegoMT2's 435 languages.

[1] No Language Left Behind: Scaling Human-Centered Machine Translation

Nevertheless, we would like to underscore that NLLB-200-1.3B only supports 200 languages. Our contribution to utilizing NLLB includes effort in extending the vocabulary from 200 languages to 435 languages. The tokens for an additional 235 languages are initiated randomly, resulting 0.3B extra parameters compared to NLLB. In this way, we can best utilize the pre-trained parameters in NLLB while largely expanding the language support in LegoMT2, in a cost-effective way.

Thank you very much for your insightful feedback and suggestions. Please see our responses to each of your comments listed below.

Weakness: The paper did not conduct specific verification experiments on parameter interference to demonstrate that the performance improvement of LegoMT2 over finetuned NLLB-200-1.3B indeed stems from the alleviation of parameter interference phenomena.

Parameter interference is a fundamental problem in multilingual machine translation.

It refers to the competition between different languages for the limited parameters of a model when we hope to use a single model to handle all translation directions. This can result in good translation results for some languages, while the translation results for other languages may be less satisfactory.

As illustrated in Table 1, directly tuning NLLB-200-1.6B yields results that are inferior to those of NLLB-200-1.3B. However, LegoMT2 effectively enhances performance across over 400 languages, which suggests that LegoMT2 successfully mitigates parameter interference.

Question 1: Which of Single-FT or Single-FT + MoE in Table 3 is used for the experiments in Table 1 and Table 2? Have the translation performance of both been evaluated?

The Single-FT is used for experiments in Table 1 and Table 2. We evaluated a pretrained 12B model (M2M-100-12B) but did not fine-tune the 12B model, because the training 12B model is too slow (16x slower in Table 3). It would require 240 days on 64 A100 GPUs to complete the training, which is prohibitively costly.

Question 2：Have any other methods for MERGE operation of non-blocking federated learning, apart from simple averaging, been tried and evaluated?

No, we do not try other methods for MEREG.

1. At present, this MERGE operation is adequate for LegoMT2. The existing MERGE operation, which involves replacing the global module on the client side with the average of the parameters obtained from the server, is the standard merge operation in federated learning.

2. Testing different merge operations is a very resource- and time-consuming operation. While we could explore more variations, it’s important to note that LegoMT2 represents a substantial pre-training effort. LegoMT2's training takes 15 days on 64 80G A100 GPUs.

3. We sincerely hope that you reconsider the value of our paper. In this paper, we proposed a novel training framework, LegoMT2, designed for massive massively multilingual machine translation (MNMT) systems.

• We introduced an efficient training framework for MNMT that supports 435 languages, which is more than any other system.
• LegoMT2 partitions the model and data together and employs an efficient non-blocking algorithm to accelerate training.
• LegoMT2 is small but powerful, which works only with 1.6B parameters and achieves better results than other models at the same or larger size (only behind NLLB-54B which is 30x larger).

Question 3： How about LLMs for Multilingual Machine Translation？

The comparison with ChatGPT is in the Appendix. Both in the En→X and X→En direction, ChatGPT falls behind LegoMT2 even with eight-shot.