The Reasonableness Behind Unreasonable Translation Capability of Large Language Model

Q6: The 'bloomz' dataset, which is open-source and includes translation tasks, doesn't seem to have been addressed in the paper. LLMs are trained on vast amounts of data, so there's a need to check for (partial) overlap with the test set.

ANSWER: Thanks for your advice. We need to clarify that our experiments do not use the BLOOMZ model or its instruction tuning dataset (P3 or xP3). Instead, we use the BLOOM model trained on ROOTS corpus[6] which does not explicitly contain parallel corpus. ROOTS corpus is only partially publicly available so it would be infeasible to accurately compute the overlap between the BLOOM pre-training data and the WMT test set. Therefore, we randomly sample $50$ cases from WMT21 EN-ZH and ZH-EN respectively and estimate the overlap proportion by manually searching similar passages in the pre-training corpus with the ROOTS fuzzy match tool[7]. Following previous work[3], we regard a test case as leaked if a substring of its target sentence with a minimum length of $15$ tokens is contained in the top-10 search results returned by the ROOTS fuzzy search tool. Through our analysis, the overlap proportion of both WMT21 EN-ZH and ZH-EN is less than 2.0%.

We have incorporated the examination of train-test overlap in our updated version of submission (Appendix A.3) with the revision highlighted in blue.

Q7: Has there been any consideration that an excessive amount of bilingual data might dilute the pre-trained knowledge in LLM and impair its translation capability?

ANSWER: In theory, the unintentional bilingual data is not supposed to dilute the translation knowledge of LLM (but may impair the knowledge for other downstream tasks) and we can observe a promotion in translation performance after post-training on unintentional bilingual data for most experiment results in Table 4 and Table 5. Moreover, compared with the 1.6T pre-training data of BLOOM, our gathered unintentional bilingual data is rather minuscule in volume.

[1]Briakou, Eleftheria, Colin Cherry, and George Foster. "Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability." arXiv preprint arXiv:2305.10266 (2023).

[2]Vilar, David, et al. "Prompting palm for translation: Assessing strategies and performance." arXiv preprint arXiv:2211.09102 (2022).

[3]Garcia, Xavier, et al. "The unreasonable effectiveness of few-shot learning for machine translation." International Conference on Machine Learning. PMLR, 2023.

[4]Bawden, Rachel, and François Yvon. "Investigating the translation performance of a large multilingual language model: the case of bloom." arXiv preprint arXiv:2303.01911 (2023).

[5]Xu, Wenda, et al. "Instructscore: Towards explainable text generation evaluation with automatic feedback." arXiv preprint arXiv:2305.14282 (2023).

[6]Laurençon, Hugo, et al. "The bigscience roots corpus: A 1.6 tb composite multilingual dataset." Advances in Neural Information Processing Systems 35 (2022): 31809-31826.

[7]Piktus, Aleksandra, et al. "The roots search tool: Data transparency for llms." arXiv preprint arXiv:2302.14035 (2023).

Q4: There are flaws in the methodology, suggesting that the prompts might influence the results. Even a minor change in a prompt can result in different translations. In light of this paper, how do the authors view this issue?

ANSWER: Thanks for your reminder! Our choice of prompting follows previous work[2][3] and we totally agree that different wording in prompt indeed causes differences in translation performance. To investigate the variance caused by different prompting, we adopt the most competitive prompt reported in [4]. Specifically, we format a given test case and $n$ randomly samped in-context examples $(x_1,y_1), (x_2,y_2), (x_3,y_3) ... (x_n,y_n)$ into the following template.

{source_language}: {$x_1$} = {target_language}: {$y_1$}

{source_language}: {$x_2$} = {target_language}: {$y_2$}

{source_language}: {$x_3$} = {target_language}: {$y_3$}

...

{source_language}: {$x_n$} = {target_language}: {$y_n$}

{source_language}: {$x_{test}$} = {target_language}:

The experiment results on post-training BLOOM-560m are shown in the table below:

The experiment on post-training BLOOM-7.1b is shown in the table below:

As is shown in the tables, turning to a different prompt indeed brings fluctuations in translation performance, but our conclusion remains. In other words, our finding is relatively insensitive to different prompting templates.

Q5: The evaluation methods have their limitations. If possible, please consider using the Instruct Score from EMNLP 2023

ANSWER: Instruct Score[5] is a brilliant idea providing a solution to explainable and learnable text generation evaluation. Following your advice, we use the instruction score to measure the translation ability, and the results on post-training BLOOM-560m are shown below:

The evaluation results on BLOOM-7.1b are shown in the table below:

We have incorporated the evaluation results (Appendix B) and a brief introduction of Instruct Score[5] (Section 4) in our updated version of submission with the revision part highlighted in blue.

We are grateful for your time and effort in reviewing our work. The constructive and insightful suggestions would definitely help us improve the work. We would like to address your weaknesses and questions one by one as below and we are looking forward to your feedback or further comments. Your support is of great importance to us.

Q1: How can we be certain that it's related to numbers and punctuation?

ANSWER: Thanks for your suggestion! Following your advice, we substitute all digits in purified bilingual data with words in respective languages (e.g., "one" for $1$ in English and so on) and repeat our experiments. The experiment results are shown in the Table below and updated in our submission (Section 5.4):

From the table, we can see that the positive mutual information on EN-ZH direction disappears once we remove all the digits in the pre-training data. Besides, we notice that the monolingual/bilingual perplexity is worse than that trained on the original purified data, possibly because replacing all the digits with English words or Chinese characters would cause a change in the pre-training data distribution. The experiment results primarily agree with our hypothesis that the existence of co-occurring digits might be instrumental in LLM obtaining translation ability from purified bilingual data. However, to draw a more definitive conclusion, more experiments are needed and we would rather leave it as a topic for future exploration.

Q2: At its core, it's an ablation study at the data level. Is the spontaneous alignment capability of large models based on the power of the attention framework?

ANSWER: Honestly, our work is indeed a sort of ablation study from the data-driven perspective, following previous work[1]. Our work provides an explanation of the underlying mechanism by identifying the existence of unintentional bilingualism at finer granularity (which is omitted by [1]) and quantitatively analyzing their effect in language model pre-training. It is risky to draw conclusions on the power of attention framework based on our experiments since the question is not the focus of our work. But would like to work on how the design of model architecture including attention mechanism affects LLM's translation capacity in the future.

Q3: Regardless, does joint training of an LLM with two or more languages always generalize to translation capabilities, given a fixed number of languages in the data?

ANSWER: Frankly speaking, we cannot assure that LLM with two or more languages will always generalize to translation ability, which requires enumerating all language pairs and thus is infeasible. Alternatively, we perform experiments on English and Chinese as two representatives since they are the top $2$ languages in the language makeup of BLOOM pre-training data and they almost share no scripts. We believe that our findings are agnostic to the specific languages and could be extrapolated to other language pairs.

We are very excited to receive your comment approving the novelty of our findings and the solidity of our experiment design. We are grateful for your time and effort in reviewing our work. We carefully read through your insightful and helpful comments and we would like to address weaknesses as well as questions one by one:

W1: This data is at the core of the argument of the paper, better quality control would make a more convincing case.

ANSWER: Thanks for your suggestion. We totally agree that data quality control is very important for our setup. Therefore, we perform a human evaluation of the quality of our collected corpus. Specifically, we gather judgments from three in-house annotators by randomly sampling $100$ cases from SA/WA/CS data collected from EN/ZH corpora respectively and asking them whether the shown texts agree with the definition of SA/WA/CS. The qualification of each case is determined by the major vote of three annotators. The annotators are volunteers from our institute, each possessing a bachelor's degree and excelling at both languages. They are unknown of the source of these sentences. The experiment results are shown in the table and updated in our submission (Appendix A.1):

Besides, to verify whether varying hyperparameters for classifying sentence alignment yield different conclusions, we modify the similarity threshold for sentence alignment classification from $10.0$ to $12.5$ and $15.0$ (measured in BLEU). Utilizing the updated sentence alignment thresholds, the experimental outcomes for post-training BLOOM-560m are displayed in the tables that follow:

As shown in the tables, alterations to the threshold may impact specific translation performance, but not our overall conclusion. In essence, our findings remain robust under varying similarity thresholds and are relatively unaffected by minor adjustments of the threshold. Your advice definitely helps to improve our work and we have incorporated your concern and our answer in our updated submission (Appendix A.2).

W2: Experiments use mostly smaller models, as well as surrogate methods, and some models are not even converged.

ANSWER: Thanks for your advice. Limited computational resources are indeed a concern when designing our experiment. However, from another standpoint, BLOOM-560m and BLOOM-7b1 are the two most widely used versions within the BLOOM series, as evidenced by download statistics from the Huggingface Hub. Consequently, we have selected BLOOM-560m and BLOOM-7b1 as representatives. Training a fully converged BLOOM-560m model necessitates 92.61 days on 32 A100 GPUs, according to BLOOM's technical report [1], which surpasses the computation budget of many institutions. As a result, our surrogate method is more replicable and we promise to release our code to facilitate relevant research.

W3: Section 5.4 would benefit from a comparison with a model using UBD data somehow.

ANSWER: Thanks for your advice! We pre-train a BLOOM-560m model on a blend of unintentional bilingual data to draw a comparison with the one trained on purified data. The results are shown as below:

From the table, the effect of unintentional bilingual data is evident. When comparing the results with Table 8 in section 5.4, it becomes evident that translation capacity acquired from the purified (uncontaminated) bilingual corpora is very weak though it does exist.

Q1: Shouldn't perplexity be lower in Table 9 (using UBD) than in Table 10 (purified)?

ANSWER: We apologize for any ambiguity. However, Table 8 and Table 9 should not be directly compared, as they differ in hyperparameters and language composition. In the experiment presented in Table 9, the data is randomly sampled from the union of mC4.en and mC4.zh, where English data prevails. Conversely, in the experiment depicted in Table 10, equal amounts of English and Chinese data are employed.

Q2: Although WA is better than WA-rand in Table 4, it is also inferior to both SA and CS, often by a large margin. Could you better support the claim that "the effect of word alignment data is comparable or even superior to that of sentence alignment"?

ANSWER: Our primary conclusions are derived from Table 4, Table 5, and Table 7. We observe that, in the EN-ZH direction for BLOOM-7.1b (Table 4), several values are lower than those for SA and CS. However, in Table 5, particularly in the EN-ZH direction, WA is comparable to or even exceeds SA. Consequently, we deduce that WA also plays a significant role in the acquisition of translation ability.

Q3: Additional point in Section 5.6 and other misc/typos.

ANSWER: Thanks for the advice! we have incorporated your advice into our updated version of submission with the revision part highlighted in blue.

[1]Workshop, BigScience, et al. "Bloom: A 176b-parameter open-access multilingual language model." arXiv preprint arXiv:2211.05100 (2022).

Thanks for your time and expertise invested in reviewing our work! we are encouraged to receive the recognition of extensive experiment and analysis. We carefully read through your constructive and valuable suggestions and we would like to answer your questions one by one as below. We hope we can address your weaknesses as well as questions and we would be grateful for your support, which means a lot to us.

W1：The authors focus on English and Chinese data to identify unintentional bilingualism types, and I was wondering if any other bilingualism types exist when you check the other languages.

ANSWER: Thanks for your suggestion. Our work is inspired by [1], which identifies $4$ types of language contamination (unintentional bilingualism) phenomenon in more than $50$ languages. We use their subdivision of language contamination as a reference and identify three types of unintentional bilingualism in our work.

Following your advice, we perform data mining on mC4.fr searching for unintentional bilingualism between English and French. We randomly sampled 160,000 French documents and 270,000 English documents and the proportion of documents with three types of unintentional bilingualism are shown in the Table below.

We can observe that a great portion of documents contain unintentional bilingualism (between English and French). Note that our detection and mining of unintentional bilingualism is subject to the accuracy of the language detection tool. We primarily chose English and Chinese data as two representatives because of their high proportion in the pre-training corpus of BLOOM. Meanwhile, it is technically easy to detect unintentional bilingualism since they almost share no common scripts.

W2: However, the question still remains on how to effectively enhance translation capability further against the supervised translation models.

ANSWER: Thanks for your concern. Enhancing the translation capacity is crucial to the downstream application of large language model. However, our work's primary objective and contribution lie in tracing the origin and mechanisms of LLM's translation ability from a data-driven standpoint. As a by-product, we observe that post-training on unintentional bilingualism typically results in improved translation performance. Motivated by this phenomenon, one potential approach could involve incorporating more bilingual or cross-lingual data into the pre-training corpus or the supervised fine-tuning corpus, as explored by [2]. Consequently, we believe our findings will inspire researchers within the community to further enhance translation ability.

Q1: In Table 3, why is #seq smaller than #Doc? The paper describes "We concatenate all the sentences in documents to form input sequences of fixed length" and I assume that #seq would be larger in #Doc.

ANSWER: Sorry for the ambiguity. We actually concatenate all the unintentional bilingual data (SA/WA/CS) together with their nearby context in documents to form an input sequence of fixed length. So it is possible that the unintentional bilingual sentences from multiple documents constitute a single input sequence of fixed length ($1024$ in our experiment). We have corrected the caption in the updated version (highlighted in blue) to avoid misunderstanding.

Q2: Rephrasing and correcting typos

ANSWER: Thanks for your advice. We have incorporated your advice into our updated version of submission with the revision highlighted in blue.

[1]Blevins, Terra, and Luke Zettlemoyer. "Language contamination helps explain the cross-lingual capabilities of English pretrained models." arXiv preprint arXiv:2204.08110 (2022).

[2]Zhu, Wenhao, et al. "Extrapolating Large Language Models to Non-English by Aligning Languages." arXiv preprint arXiv:2308.04948 (2023).

We are grateful for your acknowledgment that our work sheds light on the amazing translation ability of large language models. We are encouraged by your recognition. We read through your insightful comments, which definitely help us improve the work. We would like to address your weaknesses as well as questions one by one as follows:

W1: We just don't know how important they are and is there any other reason (e.g. the presence of other stuff) that is actually even way more important than the three?

ANSWER: Good question! We recognize that there may be additional factors within pre-training data, beyond the three types of inadvertent bilingualism, that contribute to translation capabilities. The experiments in Section 5.4 confirm the presence of such factors in the purified data. However, compared to unintentional bilingualism, these other factors have a relatively minor impact. As illustrated in the table below, we post-train BLOOM-560m using either a blend of the three types of unintentional bilingual data or the "residual data" sampled from the complementary portion of unintentional bilingualism within the entire pre-training corpus to evaluate their respective effects. (Note that residual data does not equal purified data, since the former may contain unrecognized unintentional bilingualism.) To ensure a fair comparison, the "residual data" utilized in our experiments is consistent with the unintentional bilingual data in terms of size and language makeup.

From the table, it becomes evident that the influence of other factors pales in comparison to that of unintentional bilingualism. Hence, we can deduce that unintentional bilingualism is the most predominant factor, surpassing the others in effect.

---

W2: There are no data points on why training on purified training data would lead to translation ability, just some postulating about the reasons and that is it.

ANSWER: Thanks for your advice. Training on purified training data would lead to relatively weak translation ability. This is verified by our experiments: the translation ability is only partially observed in the EN-ZH direction, but it is not clear in the ZH-EN direction, as shown in Table 8. As a result, pinpointing the origin of such a weak translation capacity is very challenging because of its weak signal. Furthermore, the experiment referenced in our response to W1 also indicates that the LLM's translation capacity primarily stems from unintentional bilingualism, while the remaining data is of less importance. Therefore, focusing on unintentional bilingualism instead of purified data would be more valuable to understanding the translation ability of large language models. However, to render our analysis more comprehensive and self-contained, we substitute all digits in purified bilingual data with English words or Chinese characters (e.g., "one" for $1$, "two" for $2$ in English and "一" for $1$ in Chinese) and repeat our experiments, following the advice of reviewer 6ach. The experiment results are shown in the Table below and are updated in our submission (Table 9):

From the table, we can see that the positive mutual information on EN-ZH direction disappears once we remove all the digits in the pre-training data. Besides, we notice that the monolingual/bilingual perplexity is worse than that trained on the original purified data, possibly because replacing all the digits with English words or Chinese characters would cause a change in the pre-training data distribution. The experiment results primarily agree with our hypothesis that the existence of co-occurring digits might be instrumental in LLM obtaining translation ability from purified bilingual data. However, to draw a more definitive conclusion, more experiments are needed and we would rather leave it as a topic for future exploration.