Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM

Thanks for the positive and detailed feedback.

1. Unfortunately the sizes of PaLM2 models are not public. Barring the differences in model size, however, the gap between PaLM2 and Gemma is also driven by differences in their pre-training recipes (e.g., PaLM2 are highly multilingual LLMs, Gemma models are more monolingual); as well as the fact that PaLM2 are relatively older models than Gemma. We speculate PaLM2 XXS to be comparable to NLLB 1.3B due to their similarity in compute requirements. To roughly align these models nonetheless, we suggest comparing their baseline performance in Table 2.

2. Admittedly, mufu-finetuned models have substantially higher latency than NLLB, which are encoder-decoder models trained specifically for translation. Low-resource languages are however notoriously difficult for LLMs [Robinson et al., 2023; Zhu et al., 2024] due to the huge disparity in resource levels between languages. We now further elaborate this point in lines 500 and 535.

3. Thanks for pointing out the possibility of overfitting in the other models. Despite seemingly superior performance of mufu20lora based on mean chrF, we find lora to be worse than full finetuning in translations to very-low-resource languages (Figure 2a). In Table 2, PaLM2 XS and PaLM2 S finetuned with the baseline method overfit and perform worse than PaLM2 XXS. This is not the case for Mufu finetuned models, as we see improvement with increasing model capacity. It is also possible that the model overfits to translations in HRLs, but not in LRLs — in which case, a reasonable approach might be to terminate HRL training early (i.e., a form of curriculum learning). We leave this experiment to future work, and now address the point in Section 4.3, line 426.

4. We conducted most experiments with PaLM2 XXS–NTL as it was the best performing model. Should the work be accepted we will include the corresponding ablation results for the best performing Gemma model in the Appendix.

5. Thanks for suggesting to compare the distilled mufu20 model against a vanilla distillation setup. We note, however, that the current teacher model (PaLM2 S) is poor in translations to low-resource languages. We are nonetheless happy to consider other teacher models, and include this result in the Appendix of the final manuscript if required.

6. Thanks for raising that there might be accidental translations in the wrong language. We manually inspected a number of Indonesian and Chinese languages, and found no absolute incorrect language in the translations. It is however difficult to assess the accuracy systematically as many of these languages share the same scripts, have similar vocabulary, and borrow words from one another (see for example, Table 4). For other language pairs with <20 chrF (e.g., Fon, Tamasheq, Tigrinya), we could only perform sanity checks for accidental translation in the wrong script, as we are unfamiliar with these languages.

7. Win rate is the percentage of language pairs where the model outperforms a benchmark. For example, NLLB 54B MoE outperforms the teacher in 113/201 ≈ 56.2% language pairs based on chrF; whereas PaLM2 S finetuned with mufu20lora outscores the teacher in 199/201 ≈ 99% language pairs. We now clarify this in Section 4, line 197.

8. Thanks for raising that lines 207-211 could be further improved. In Mufu we finetune the models against the gold-standard translations, and expect improvement from the postediting targets. This is effective for LRLs with low-quality postediting candidates. However, improving high-quality translations in HRLs is harder and requires the student model to also learn the subtle differences between model- and human-generated output. It is also possible that the LLM teacher surpasses human for some translations in HRLs, in which case, learning from the human output could be detrimental. We now elaborate this cause of decline in performance relative to teacher in Section 4, line 263.

Thank you for the questions.

1. NLLB 1.3B was distilled from NLLB 54B MoE. The latter was trained in hundreds of translation directions, on large-scale mined bitext and monolingual data augmented with backtranslation [1]. Thus it is not surprising that PaLM2 XXS requires further NTL pretraining and gains from Mufu to achieve comparable performance.

2. Thanks for raising that the statement needs further support. LLMs closely resemble human translations in their use of lexical and linguistic features [2]. LLM output also becomes increasingly difficult to be distinguished from human translations [3], given the current evaluation method that identifies similar errors in both systems [4]. We now include these citations following the claim in line 265. More importantly, however, we note in the manuscript that the key reason for decline is that finetuning on human-generated translations is suboptimal compared to finetuning on high-quality model-generated data [5].

[1] Costa-jussà, Marta R., et al. "No language left behind: Scaling human-centered machine translation." arXiv preprint arXiv:2207.04672 (2022).

[2] Sizov, Fedor, et al. "Analysing Translation Artifacts: A Comparative Study of LLMs, NMTs, and Human Translations." Proceedings of the Ninth Conference on Machine Translation. 2024.

[3] Kocmi, Tom, et al. "Findings of the WMT24 general machine translation shared task: the LLM era is here but mt is not solved yet." Proceedings of the Ninth Conference on Machine Translation. 2024.

[4] Zhang, Ran, Wei Zhao, and Steffen Eger. "How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs." arXiv preprint arXiv:2410.18697 (2024).

[5] Finkelstein, Mara, and Markus Freitag. "MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods." The Twelfth International Conference on Learning Representations.

Thanks for the positive feedback.

1. Our analyses include 201 languages spoken in 19 regions (e.g., Middle Africa, Central Asia, Northern Europse, etc.), of which 113 languages are considered low- and very-low-resource. While this is only a fraction of 7000+ languages in the world, our setup is unfortunately limited by the availability of high-quality parallel datasets in the other languages. However, we will consider expanding the set of languages in our analysis of self-attention (Section 4.2), and including in the final draft analyses for other languages analogous to Figure 3 in the Appendix.

2. We highlighted the contributions of auxiliary languages by comparing mufu5 against mufu5hrl, which consists of 5 HRLs (Dutch, Russian, French, Chinese and Spanish) as auxiliary languages; and mufu5tr, which removes the postediting target in the context. We showed mufu5 to be superior to both mufu5rhl and mufu5tr, indicating the importance of the relevance of auxiliary languages to the target language and postediting candidates in the context. In Section 4.2, we further corroborated the impact of auxiliary candidates, showing mufu finetuned models to be capable of inferring correct translation from relevant languages by cross-lingual lexical alignment in multihead attention. Some languages are missing in URIEL, we therefore include random auxiliary languages in context. Our preliminary analysis with controlled language resource level show target languages with related auxiliary candidates to have on average higher chrF improvement ratio ((chrF_mufu - chrF_baseline) / chrF_baseline) across models and mufu{5, 10, 20}. We will consider adding in the final manuscript an ablation with the inclusion of random auxiliary candidates for all target languages if required.

3. Thanks for this suggestion. We will report in the final manuscript the translation performance of the distilled models on NTREX with 5-shot prompting to improve cross-domain adaptability.

Thanks for the review and suggestions.

1. We reported the initial prompt selection in Appendix A.1—unfortunately we are unable to move the description to the main text due to page limit. We have tried mufu20+5hrl, which includes 5 HRLs candidates in addition to the same set of auxiliary languages as mufu20, and found it to be less performant than the latter. This result is now reported in Table 2. We are also happy to consider any other suggestions on the mix of auxiliary languages for further ablation studies.

2. We primarily report our results in chrF rather than BLEU as the latter heavily relies on tokenization that is underdeveloped for many low-resource languages. Nonetheless, we now report BLEU scores in Appendix A.4, which are consistent with the positive results shown in Table 2 in the main text. Prior to mufu finetuning, PaLM2 XXS–NTL has been continued-pretrained on a corpora derived from Next-Thousand-Language effort (Caswell et al., 2020), which contain monolingual and parallel sentences in 1000+ languages. This results in significant improvement of translation performance when the model is finetuned with mufu prompts, as compared to PaLM2 XXS. Given the gap in performance, we speculate further improvement as well in other models with similar monolingual training.