Why Not Transform Chat Large Language Models to Non-English?

Thank you for the positive comments and constructive suggestions.

W1: The experiments are primarily conducted on transforming LLaMA-2-chat-7B to Thai, which may limit the generalizability of the findings to other non-English languages and other models...

Please refer to the responses for all reviewers.

W2: While the proposed method exhibits promising results in MT-bench and AlpacaEval, it is not tested on other traditional NLU benchmarks, just like MMLU in English...

Thanks for your constructive suggestion. We further evaluate TransLLM for causal commonsense reasoning in Thai using the XCOPA dataset [1] and outperform strong baselines including ChatGPT. Detailly, XCOPA consists of 500 questions. We conduct experiments in the zero-shot setting. Since chat LLM may output results in different formats, we evaluate the results by one human annotator instead of simply matching. As shown in the following table, TransLLM outperforms ChatGPT by 8.38%. The performance of Typhoon and X-LLaMA is even worse than random guess (50%). That is because these models sometimes can't follow the instructions and output hallucinations.

Accuracy of different models in TH on XCOPA:

Q1: Please address the weaknesses above.

We have tried our best to address the weaknesses. We hope the above response could address your concerns.

Q2: For the construction of the TCOT, did you try other concatenation templates besides the one used in the paper? How sensitive is the performance to the choice of the template?

Following the designs for the template of in-context learning [2], we consider several aspects when we design the template: 1. Ordering. TCOT is inspired by human beings. It is intuitive to first translate the question from the foreign language to the native language, answer in the native language, and then translate the answer to the target language. 2. Formatting. To build the relationship between sub-tasks and TCOT, we use different special tokens to represent different tasks across different procedures. We do not use the natural language template (NLT) as in PLUG because the template will be trained many times across different training steps, which could significantly affect its original meanings. We conduct experiments using the NLT in PLUG (Let me interpret the instruction in English:...Then the English response is:...Final, the Thai response is:...). The following results show that the proposed template is better than NLT.

Comparison between vanilla TransLLM and TransLLM w/NLT on MT-bench in Thai under GPT-4 evaluation:

Q3: Table 9: How was the human evaluation carried out? How many annotators were involved, and what is the inter-annotator agreement? The final scores are all integers, so it is not clear how to final scores were calculated.

We provide the details of human evaluation in Appendix A.4. We obtain the win/tie/loss tag by comparing evaluation scores of different models since pair-wise comparison significantly improves the evaluation cost. More importantly, as shown in Table 5 of [3], there is a high agreement between direct evaluation and pair-wise comparison. Due to the annotation cost, we only conduct human evaluation once for each response. To address this concern, we further re-evaluate 50% of these results following the same procedure and provide the inter-annotator agreement as follows. There is a high inter-annotator agreement in our evaluation.

Inter-annotator agreement on MT-Bench in Thai:

Typos

Thanks a lot! We will double-check our manuscript and further revise the paper carefully.

[1] Ponti et al., 2020. XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning.

[2] Dong et al., 2020. A survey on in-context learning

[3] Zheng et al., 2024. Judging llm-as-a-judge with mt-bench and chatbot arena.

Thank you for the positive comments and constructive suggestions.

W1: Flexibility of Methodology

1. Transfer requires resources: Knowledge transfer across languages relies on semantic alignment, which inevitably needs some translation resources. As we discuss in the related works, most existing works also rely on parallel corpora and external translation models for data construction (e.g. PloyLM, X-LLaMA, PLUG).
2. Required resources are acceptable: With the development of machine translation, there have been more and more parallel data for many low-resource languages. However, there are only a few instruction-following data for limited languages. As shown in Table 6 of ablation studies, large amounts of parallel data only contribute a small improvement and therefore it is not necessary for extremely low-resource languages. This is consistent with the phenomenon in LLM translation studies. Moreover, we only need small-size TCOT data for fine-tuning (50K in our experiments). So, it is acceptable to conduct human translation to generate TCOT data if there are no high-quality machine translation systems for the target language.
3. Alternative resources: For the extremely low-resource languages, we could use unsupervised or semisupervised machine translation methods (e.g. iterative back translation and multilingual pre-training) to obtain feasible translation resources.

W2: Scope

Please refer to the responses for all reviewers, where we discuss the generalizability of TransLLM and validate it in Arabic.

W3: Effectiveness

Please refer to the responses for all reviewers, where we conduct experiments on the recently released LLaMA-3-instruct-8b.

W4: Quality of MT data

Yes, the impact of machine translation (MT) quality is an interesting point. To investigate that, we also use the NLLB to construct TCOT data. We use the wmt22-cometkiwi-da to evaluate the quality of translations without references. As shown in the following table, Google Translate generally performs better than NLLB, which is consistent with the comments of our human translator.

CometKiwi score on TCOT data:

We further train a TransLLM model using the TCOT data generated by NLLB and provide the results in the following table. We can find that TCOT data with higher MT quality achieve much better performance. That is because translation ability is important for successful TCOT, while low-quality TCOT data can introduce lots of translation errors.

Comparison between different TCOT data (Google Translate vs. NLLB) on MT-bench in Thai under GPT-4 evaluation:

The impact of machine translation (MT) quality for AdvBench has been considered in human evaluation. Yong et al. [1] define the UNCLEAR label for nonsensical responses caused by low translation quality or LLM hallucinations.

W5: Typos

We will double-check our manuscript and further revise the paper carefully.

Q1: This paper mainly focuses on transferring to a single target language. Is it applicable for transferring to multiple target languages? If so, how to balance the data for multiple target languages?

Thanks for the constructive question. TransLLM is easy to extend for multilingual settings and benefits from the language IDs. mBART shows that the language IDs work well for models to distinguish tasks of different languages. The major problem, as you mentioned, is how to balance the data for multiple target languages, since the resources across languages are imbalanced. Lample et al. [2] simply smooth the distribution using a multinomial distribution. Zhu et al. [3] formulated scaling laws to optimize data allocation. We would like to explore these strategies in the future.

Q2: Line 170: A random strategy was used for data mixing. Would other mixing strategies affect the results?

The recency bias (catastrophic forgetting during training) causes the model to perform worse on tasks that have not been recently trained [4]. A straightforward and practical solution to balance the performance of different TCOT sub-tasks is to expose them equally frequently. This strategy works well in our setting.

Q3: Line 131: Since the authors mentioned being inspired by mBART, is the special language ID independently occupying a token in the vocabulary?

Yes, we add "<EN>", "<TH>" and "<RESPONSE>" as special tokens into the tokenizer.

[1] Yong et al., 2023. Low-Resource Languages Jailbreak GPT-4.

[2] Lample et al., 2019. Cross-lingual Language Model Pretraining.

[3] Zhu et al., 2023. Extrapolating large language models to non-english by aligning languages.

[4] Lyu et al., 2023. Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation.

Thank you for your time and comments.

W1: The experiments are done on only one LLM and on only one language (Thai). This severely constrains the extend to which the results could be verified.

Please refer to the responses for all reviewers.

W2: The novelty of the proposed method is very thin and very limited analysis is done to motivate that novelty.

1. Novel target: In this paper, we try to transform chat LLMs, instead of base LLMs in previous works, from English to non-English. These chat LLMs are specifically optimized for advanced abilities (e.g. multi-turn conversation and human preference alignment), making them not only more effective but also safer than their base versions. Transferring these abilities to non-English will improve the experience for non-English users and, more importantly, improve safety in non-English.

2. Novel challenges: While pre-training data for base LLMs is typically available, fine-tuning data—especially for advanced chatbot capabilities—is often lacking. Transferring these advanced abilities in chat LLMs without their specific supervised data presents a significant challenge. During the transformation of base LLMs, instruction tuning can be conducted either concurrently or subsequent to transfer training. However, in chat LLMs, advanced capabilities are integrated into model parameters prior to transformation, potentially leading to catastrophic forgetting of these abilities during subsequent transfer training.

3. Novel framework: To tackle these challenges, we introduce the TransLLM framework. For the first issue, TransLLM leverages the Translation Chain-of-Thought (TCOT) to divide the transfer process into some common tasks, whose datasets are often publicly available. For the second issue, we employ low-rank adaptation (LoRA) to preserve the original LLM parameters and recovery knowledge distillation (KD) to recover the original abilities.

4. Corresponding analysis: In our ablation studies (Table 6), we demonstrate the importance of enhancing TOCT's sub-task performance for effective transfer. Our findings in Table 7 confirm that transfer training induces catastrophic forgetting of original knowledge. Moreover, despite the widespread use of high-quality GPT-4 KD data, it fails to aid in recovering original knowledge, whereas our proposed recovery KD method achieves this successfully. Importantly, we introduce how LoRA + recovery KD mitigates catastrophic forgetting (lines 323-330). The recovery KD enables the LLM to learn a “shortcut” that uses the English knowledge from the original parameters and the new knowledge from the LoRA parameters. The experiment results indicate that we successfully transfer advanced abilities from English. TransLLM outperforms strong baselines on the multi-turn conversation benchmark in Thai without using any multi-turn conversation datasets. Notably, TransLLM outperforms GPT-4 on the safety benchmark in Thai without using any safety or human preference alignment datasets. We hope this work can lay a solid foundation for developing safe LLMs in non-English.

Q1: Tables 4 and 5 show a large gap against GPT-4, yet GPT-4 KD is not as effective? Why is this the case?

In Table 7, it is evident that transferring knowledge leads to catastrophic forgetting of the original knowledge, and GPT-4 KD fails to recover this original knowledge. The publicly available GPT-4 KD dataset comprises only 52K instances, limiting its scope despite its high quality. In contrast, the original large language model (LLM) contains a broader range of English knowledge across various domains. Thus, recovering the original knowledge can significantly enhance the LLM's capabilities. While the quality of recovery KD is lower than that of GPT-4 KD, it effectively recovers the original knowledge (Table 7). The recovery KD data can be fitted easily using the original parameters. This enables the LLM to learn a “shortcut” that uses the English knowledge from the original parameters and the transfer knowledge from the LoRA parameters. In lines 323-330, we show TransLLM successfully learns this “shortcut” by LoRA + Recovery KD.

Thank you for your time and comments.

W1: To be honest, I am not sure whether this paper has good starting point. They focus on a scenario where you want to further fine-tune the chat model in English to a chat model in other languages. Could you provide more concrete application scenarios in your introduction? I am not convinced because typically, in practice, people will use all the data available to tune a base model, and usually, this renders the best downstream task performance, or at least, the robustness defined as the average downstream task performance of a particular tuned chat LLM. Why cannot we follow this paradigm?

A high-quality chat LLM relies on high-quality training data, which is usually unavailable even in English. For example, LLaMA-2 models release parameters but withhold their valuable datasets, such as multi-turn conversation and human preference data. Acquiring such high-quality data from scratch and conducting complex training, like RLHF, in non-English languages, is notably costly. Therefore, this paper explores transferring abilities from English to non-English for chat LLMs leveraging limited available datasets, such as target language monolingual datasets, parallel datasets, and instruction tuning datasets.

In our experiments, we demonstrate that solely using all available data in the target language to continue tuning the LLM (i.e. Typhoon [1]) results in less satisfactory performance. In contrast, the proposed TransLLM effectively transfers capabilities from English. For instance, TransLLM surpasses strong baselines on the multi-turn conversation benchmark in Thai without using any multi-turn conversation datasets. Notably, TransLLM surpasses GPT-4 on the safety benchmark in Thai without using any safety or human preference alignment datasets. We hope TransLLM can lay a solid foundation for developing safe LLMs in non-English.

W2: I suggest the authors to further polish the paper abstract. I am getting confused several times, for example, in your statement ``Transforming English-centric LLMs to non-English has been identified as an effective and resource-efficient method'', why can you simply say this is a resource-efficient method? As far as I know, if you just use knowledge distillation by distilling the data synthesized from a stronger model in your target languages, you might still need to use stronger model, usually those closed-source models, e.g., ChatGPT, this is still expensive, right? Besides, as even the knowledge distillation, in the SFT stage, you have to mix all the data to do the training as well. So it is still expensive.

To enhance the non-English ability, a simple approach is that we create non-English data as much as English and perform heavy training. However, it is almost impossible due to the huge cost. Compared to this paradigm, transforming English-centric LLMs to non-English is much more efficient. The knowledge distillation (KD) data in English is from the publicly available Alapca dataset. The KD can be directly translated into the target language using available machine translation systems as shown in [2, 3]. Therefore, we can achieve KD data at a low cost. In practice, there is only about 52k KD data that needs to be tuned. This computational cost is usually acceptable.

Thanks for your suggestion. We would like to clarify these points in the paper and further polish the abstract.

[1] Pipatanakul et al., 2023. Typhoon: Thai large language models.

[2] Zhu et al., 2023. Extrapolating large language models to non-english by aligning languages.

[3] Zhang et al., 2024. Plug: Leveraging pivot language in cross-lingual instruction tuning.