Extrapolating Large Language Models to Non-English by Aligning Languages

W1: The proposed work is only compared to other LLaMA-based model

The comparison with other llama-based models aims to avoid the influence of different base models on the comparison results. To attain a more meaningful comparison, it might be more insightful to compare different data combinations (as shown in the table below). Such comparisons highlight the significance of our method: compared to using only English Alpaca data, both the Chinese Alpaca data and translation data contribute to the improvement. The combination of these datasets achieves the highest performance.

W2: Quantify the improvement in alignment by measuring the distance between the representations of positive/negative samples

Thank you for your insightful comments. Following your advice, we use English and Chinese as an example and measure the cosine distance between positive pairs and negative pairs across different layers (0, 8, 16, 24, 32). By 'positive pair', we refer to an English sentence and its corresponding Chinese translation. By 'negative pair', we mean an English sentence and a randomly sampled Chinese sentence. As shown below, in the representation of x-LLaMA, positive pairs stay closer than in Alpaca, especially in the latter half of the model layers.

W1: There is a lack of insightful discussions and analyses regarding scaling law

Thank you for your advice. We designed the current scaling law based on three motivations, which are discussed in section 3.2. In our preliminary experiments, we also explored different formula design options, and eventually settled on the best solution. In the next version, we can expand the discussion on this topic, providing more insightful analyses.

We can also explain the computation details regarding language similarity here. Our computation process aligns closely with that of Pan et al. Initially, we encode multi-lingual parallel data using LLM, and average the representations from the final layer to derive sentence vectors. To illustrate with the calculation of English-Chinese language similarity, we use the representations of English sentences to retrieve Chinese sentences. A retrieval score can be determined based on the ranking of the target sentence in the retrieval results. By averaging all retrieval scores, we can obtain the language similarity between English and Chinese. The higher the similarity, the closer the internal representations of parallel sentences within the LLM.

W2: It remains unclear if the conclusion still holds for other LLMs.

To address your concern, we implemented our methods on Pythia-6.9B [1], another English-dominant LLM. Below, we present experiment results on Chinese tasks. The conclusion mirrors our previous findings: both Chinese Alpaca data and translation data prove to be beneficial for enhancing the Chinese ability of the LLM.

[1] Biderman et al. Pythia: A suite for analyzing large language models across training and scaling.

Q1: How is the similarity of language computed, and how are the values of α and β estimated for the scaling law?

For the process of computing language similarity, please refer to our response to W1. As for the process of computing α and β, we can explain it here: given a set of {scale, performance} data points, we use a Python package to automatically estimate α and β. The values derived in this manner allow the scaling law curve to best fit the known data points.

Q2: Further elaboration on how the optimal allocation of data is obtained.

We obtain the optimal allocation by solving a non-linear programming problem, as introduced in Section 3.3. The optimization solution is achieved using the Python package scipy.optimize.

The underlying insight is: during this process, the data is preferentially allocated to the language with greater marginal benefit (the gradient of the scaling law, i.e., the performance improvement that can be brought by adding a unit of data). When each unit of data is allocated to a language to maximize its benefits, the maximum total benefit is achieved (which is reflected in the average multi-language translation performance).

W1: The employed method lacks novelty

The reviewer might overlook the following technical novelty in our paper.

1. We introduce multi-task instruction-tuning as a strategy to extrapolate LLM's english ability to non-English.
2. We quantify the scaling law in cross-lingual instruction-tuning and use these laws to optimize data allocation in a resource constrained setting.

W2: The method lacks testing on high-resource languages closely related to English, as well as on low-resource languages.

In our experiments, we do have results for low-resource languages, such as Hindi. We acknowledge that we did not consider languages closely related to English, and we will include relevant results in our next version.

W3: The method has not been sufficiently tested on multilingual NLP tasks, including reasoning tasks such as MGSM and XCOPA, as well as NLG tasks like XLSum.

Thank you for your suggestion. We report results on additional tasks below, including knowledge assessment (mLAMA), summarization (XLSum), and reasoning (mGSM). The experimental phenomena on mLAMA and XLSum are similar to previous experiments. mGSM is an exception here, as all models perform poorly on this task. We believe this dataset poses a significant challenge, and further exploration may be needed to improve model reasoning capabilities and facilitate the transfer of reasoning abilities across different languages.

Q1: Is the translator engine effectively aligned with human-annotated data in terms of quality?

We understand your concern. The translator we utilize is a high-quality commercial translation engine. To check the quality of the data, we sampled and manually checked a portion of it, and found that the translated Alpaca data is accurate and reliable.

Q2: It would be beneficial to analyze to assess the significance of these prompts and their impact on the overall performance.

The depiction in Figure 1 is merely for illustration. In reality, during the instruction-tuning process, we consistently use the Alpaca template [1] to wrap the instruction, input and output - a common practice in this field. We adopted this method in all our experiments, thus holding this experimental variable constant. Therefore, we believe it won't influence the conclusions drawn from our experiments. [1] Taori et al. Stanford Alpaca: An Instruction-following LLaMA model.

Q3: Conducting ablation analysis would contribute to a more comprehensive understanding of the overall training process.

Following your advice, we conducted experiments to demonstrate the value of both Chinese Alpaca data (Alpaca-Zh) and English-Chinese translation data (En-Zh). Compared to using only English Alpaca data, both the Chinese Alpaca data and the translation data contribute to improvement. The combination of these datasets achieves the highest performance.

W1: The current evaluation setup is not fair enough, preventing the production of effective conclusions.

Thank you for your comments. We acknowledge that comparing our models with Chinese-Alpaca and Bayling may cause confusion as they are not directly comparable. In our next version, we will exclude their results and instead provide a more meaningful comparison.

W2: Ablation study is needed to demonstrate the value of added data.

Following your advice, we conduct experiments to demonstrate the value of both Chinese Alpaca data (Alpaca-Zh) and English-Chinese translation data (En-Zh). Compared to using only English Alpaca data, both the Chinese Alpaca data and the translation data contribute to improvement. The combination of these datasets achieves the highest performance.

W3: the ablation results should be reported on all considered languages.

Following your advice, we update ablation results on all considered languages on MLQA dataset (results on El, Ru, Tr are not rpeorted because MLQA does not support these languages). The conclusion is consistent: both the Alpaca data in target language and translation data contribute to improvement and the combination of these data achieves the highest performance.

W4: The research questions the paper seeks to answer are not presented with clarity.

Sorry for the unclear presentation. We hope the updated results included in this response can better clarify the contributions of our paper.

W5: how can the model achieve enough proficiency to generate fluent target languages with only limited exposure to foreign languages?

While utilizing a large amount of non-English data is indeed the most direct method to enhance non-English capabilities, it imposes significant demands on data scale. Our objective is to transfer English capabilities to non-English with minimal data. This approach effectively leverages the model's English capabilities and presents a more data-efficient choice.

W6: Given that there are many high-quality corpora available, why should they not be used for translation data finetuning?

Indeed, there are many high-quality translation datasets available for experiment, but it is not feasible for us to try them all. To our knowledge, NewsCommentary and WikiMatrix are two representative high-quality translation datasets. Through our experiments on these two datasets, we aim to demonstrate that it is possible to transfer English capabilities to non-English by aligning languages. In line with your suggestion, we also present results using other translation datasets. We used the NIST English-Chinese training set and report the results below. As the results indicate, utilizing translation data from different sources does not seem to significantly impact the final results.

W7: Concern regarding the creation and evaluation of the MI-Eval dataset.

The original goal of creating the MI-Eval dataset was to evaluate a model's capability to follow multilingual instructions. However, we also acknowledge the difficulty in automatically evaluating the quality of an LLM's response. Using chatGPT for evaluation is a common approach, but it poses issues related to reliability, interpretability, and reproducibility. Therefore, we consider removing this part in the next version.

Q1: It would be good to report performance on English QA benchmarks before and after finetuning.

Following your advice, we report the performance of x-LLaMA on the English XQUAD below. Compared to the English Alpaca model, the x-LLaMA model that has been fine-tuned on different languages does not indeed show much degradation in performance on English tasks.

Q2: Difference between MI-Eval and Alpaca dataset

We created the MI-Eval for evaluation instead of using Alpaca for two reasons: 1. Alpaca has been used for training, and using training data for testing may not reflect the true generalization ability of the model; 2. Alpaca is an English dataset, making it unsuitable for evaluating non-English capabilities.