Thanks for your time and constructive comments.

Weaknesses 1: Lack of a baseline that directly employs multilingual supervised fine-tuning (MSFT)

Thank you for your valuable suggestion. We have conducted MSFT on Llama3.1, and the results are presented in the following tables. Multilingual TransLLM (Llama-3.1 to Multilingual in table 5 and 6) demonstrates significantly superior performance compared to MSFT across five languages.

Comparison between multilingual TransLLM and MSFT on MT-bench:

Comparison between multilingual TransLLM and MSFT on Aya-dataset:

Furthermore, we would like to emphasize that similar evidence has been presented in previous results. In previous experiments, we included supervised fine-tuning baselines for a single language (SSFT). As demonstrated in both prior work (Zhu et al., 2023) and this study, MSFT only achieves performance comparable to SSFT. More importantly, we incorporated state-of-the-art open-source baselines such as Typhoon for TH and AceGPT for AR, and TransLLM significantly outperforms these baselines.

Weaknesses 2: Lack of an ablation experiment to demonstrate the necessity of maintaining the original model parameters until the entire training process is complete

The absence of such an ablation model has been addressed. We conduct the experiment and present the results in the table below. When using LoRA during fine-tuning but not during pre-training, there is also a noticeable performance degradation. However, it still outperforms TransLLM without LoRA throughout the training process. This underscores the importance of preserving the original model parameters until the training process is fully completed.

Comparison between TransLLM and the ablation model w/o LoRA pre-train:

Question 1: Lack of implementation details about baseline NLLB-bridge

We use NLLB-3B as a bridge during the inference stage by simply translating the query and response. Specifically, given a query $q^{\beta}$ in target langugae $\beta$, the NLLB-3B model first translates it into English as $q^{EN}$. The original chat LLM (LLaMA2) then generates an English response $a^{EN}$ based on $q^{EN}$. Finally, $a^{EN}$ is translated back into the target language $\beta$ using NLLB-3B, producing the final response $a^{\beta}$. These details will be included in the paper.

Suggestion 1: Consider using the term "original chat LLM" or a similar alternative instead of "target chat LLM"

Thanks very much for your constructive suggestion. We will change the "target chat LLM" to "original chat LLM" carefully.

Suggestion 2: Consider conducting experiments with other open-source LLMs like Mistral and Gemma

Thank you for your suggestion. Unfortunately, due to resource constraints, particularly for pre-training, we may be unable to conduct these experiments at this time. However, we plan to explore experiments with other, potentially more powerful, open-source LLMs in the future. Additionally, LLaMA-3 introduces significant advancements in training, data, and architecture, making it substantially different from its predecessor, LLaMA-2. As such, the experiments we have conducted already demonstrate the generalizability of the proposed framework.

References

Zhu et al., 2023. Extrapolating large language models to non-english by aligning languages.

Thank you for your response! We conducted the ablation experiment "w/o LoRA pre-train" in strict accordance with your suggestion. Specifically, we conduct continual pre-training with LoRA and merge the parameters afterward to approximate full-parameter pre-training. Subsequently, LoRA is applied for fine-tuning during the transfer stage. The remaining settings are consistent with those used in the ablation experiments in Table 7. For clarity, we we would like to rename this experiment to "w/o LoRA in pre-training". We sincerely thank you for your valuable contributions to improving our paper!

Thanks for your comments again. We would like to know if our response addressed your concerns. We look forward to discussing these concerns with you.

Weaknesses 3: The paper focus on GPT-4 as the main baseline, while GPT-4 multilingual capacity is not properly documented. A multilingual LLM baseline would be more appropriate.

Thank you for your valuable suggestions. As demonstrated in both prior work (Zhu et al., 2023) and this study, multilingual LLMs generally perform on par with language-specific LLMs. Beyond GPT-4, our study already incorporates strong baselines tailored to individual languages, such as Typhoon for Thai (TH) and AceGPT for Arabic (AR), which are the state-of-the-art open-source LLMs for these languages. Therefore, our baselines already provide sufficient validation. GPT-4 serves as a valid baseline, given its well-documented multilingual capabilities as outlined in its technical reports and those of Llama-3, both of which were cited in the previous version of the paper. For a more direct comparison, we have added multilingual supervised fine-tuning (MSFT) on Llama3.1 as a baseline, with results presented in the following tables. Multilingual TransLLM (Llama-3.1 to Multilingual in table 5 and 6) demonstrates significantly superior performance compared to MSFT across five languages.

Comparison between multilingual TransLLM and MSFT on MT-bench:

Comparison between multilingual TransLLM and MSFT on Aya-dataset:

Weaknesses 4: The main evaluation is heavily focused on Thai, with limited results for other languages.

Our evaluation approach aligns with common practices in related work. For instance, Chat Vector (ACL 2024) primarily focuses on Chinese and extends its evaluation to Korean to demonstrate versatility. To thoroughly evaluate our method with limited resources, we adopt a twofold strategy: an in-depth assessment of a single language and a broad evaluation across multiple languages. Specifically, we comprehensively evaluate TransLLM in Thai (TH) across various aspects, including instruction following, multi-turn conversation, commonsense reasoning, and safety. Additionally, we extend the evaluation of instruction-following capabilities to four other languages.

Question 1: Can the proposed method applied to any LLM instead of chat LLM in particular?

Although Table 7 shows that transforming chat LLMs achieves better performance than base LLMs, our method is not limited to chat LLMs and can also be applied to base LLMs, as demonstrated in the following table.

Comparison between TransLLM w/ base LLMs and X-Llama:

If "any LLM" refers to domain-specific LLMs such as code LLMs, json LLMs, or math LLMs, it is worth noting that the MT-Bench used in our evaluation already encompasses these domains. Therefore, TransLLM is expected to perform effectively across them.

Question 2: Is there any distillation of knowledge involved in the training process?

As demonstrated in Section 4.2, transfer training causes the LLM to significantly forget its original English knowledge (Table 8). The proposed recovery knowledge distillation method distills the original English knowledge to the trained LLM. This approach encourages the LLM to learn a generalizable pattern that minimizes the influence of LoRA parameters when generating English content, thereby restoring its original English knowledge (line 457).

Question 3: Can you elaborate on the "chain-of-thought" in TCOT data format?

Chain-of-thought (Wei et al., 2022) breaks complex problems into a series (chain) of manageable steps (thoughts). In TCOT, these steps are pre-defined, drawing inspiration from how humans handle a second language. As illustrated in line 194, solving the query $q^{TH}$ in the target language TH using TCOT involves three steps: first, translating the query into English ($q^{EN}$); second, answering the query in English ($a^{EN}$); and finally, generating the answer in the target language ($a^{{TH}}$) based on the preceding steps. Special tokens are employed to guide the behavior of each step. Accordingly, the input of TCOT data is $x = q^{TH}$, the desired output is $y = \text{cat}(\langle EN \rangle, q^{EN},\langle\text{RESPONSE}\rangle, a^{EN},\langle TH \rangle, a^{{TH}})$, where $\langle\cdot\rangle$ denotes the special token defined in paper.

References

Taori et al., 2023. Stanford alpaca: an instruction-following llama model.

Wang et al., 2022. Self-instruct: Aligning language models with self-generated instructions.

Hinton et al., 2015. Distilling the knowledge in a neural network.

West et al., 2021. Symbolic knowledge distillation: from general language models to commonsense models.

Wei et al., 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.

Zhang et al., 2023. Plug: Leveraging pivot language in cross-lingual instruction tuning

Zhu et al., 2023. Extrapolating large language models to non-english by aligning languages.

Thanks for your time and comments.

Weaknesses 2: The paper is poorly written with contextually unclear terminologies (chat LLMs, Knowledge Distillation, Chain-of-thought).

We would like to address this concern first to ensure clarity in subsequent discussions.

Chat LLMs: We introduced the terms base LLMs and chat LLMs in lines 41 and 46. Base LLMs refer to foundational models pre-trained on vast corpora using next-word prediction objectives, whereas chat LLMs are derived from these base LLMs through additional post-training techniques (e.g., supervised fine-tuning and reinforcement learning from human feedback) that enable them to follow instructions and align with human preferences.

Knowledge Distillation: We introduced the concept of knowledge distillation for LLMs in lines 42 and 181. A practical approach to enabling base LLMs to follow human instructions effectively is to distill instruction-following knowledge from powerful closed-source LLMs, such as GPT-4. For example, the widely-used Alpaca dataset (Taori et al., 2023) generates queries through the self-instruct technique (Wang et al., 2022) and collects corresponding responses from ChatGPT or GPT-4. While traditional knowledge distillation (Hinton et al., 2015) focuses on learning a teacher's output distribution, knowledge distillation for LLMs, as highlighted by West et al. (2021) and Wang et al. (2022), instead relies on distilling knowledge from sampled data, since the full distribution of closed-source LLMs is typically inaccessible.

Chain-of-thought: We introduced the concept of translation chain-of-thought (TCOT) in lines 74 and 193. TCOT performs the response generation in the target language as a sequence of sub-tasks step by step: It first translates the query to English; then responds to the query in English; and finally generates the non-English answer based on all the above context.

Weaknesses 1: The paper is an engineering-focused LLMs training project without introducing any novelty or new methodology for the target problem.

Significant practical impact: The proposed TransLLM framework addresses a critical gap in transforming chat LLMs to target languages, which has not been effectively tackled in prior work.

Innovative integration and novel techniques: To enable efficient transfer, TransLLM employs TCOT to decompose the transfer process into common sub-tasks, eliminating the need for specific training data. More importantly, TransLLM improves sub-task performance through continual pre-training. To mitigate catastrophic forgetting, we propose novel the Recovery KD combined with LoRA finetuning. While components like TCOT and LoRA are established methods, their integration within the TransLLM framework is highly innovative. As noted by reviewers vJuZ and T1tU, our ablation studies demonstrate that TCOT and LoRA alone are insufficient without the full TransLLM framework.

Novel findings and insights: Our study highlights that relying on widely used GPT-4 KD can be misleading when transforming chat LLMs. Additionally, our analysis (Section 4.2) provides valuable insights into how Recovery KD + LoRA effectively mitigates catastrophic forgetting.

Superior performance and resource efficiency: Experimental results confirm the effectiveness, flexibility, and generalizability of TransLLM across diverse languages and LLMs. The framework outperforms even GPT-4 in Thai with an 8B model, showcasing its effectiveness and demonstrating a practical pathway to achieving similar results with significantly fewer resources.

Thanks for your comments again. We would like to know if our response addressed your concerns. We look forward to discussing these concerns with you.

Thank you for your valuable feedback.

Novelty. We would like to further clarify the novelty our work in the context of your concerns. While techniques like LoRA and multi-step continual pre-training are indeed well-established, our work introduces novel integrations that distinguish it from existing approaches. As noted by Reviewer vJuZ, our framework offers "detailed insights into critical decisions" through ablation studies and in-depth analysis. Each technique we utilize, including TCOT and LoRA, is motivated by the specific challenges inherent in transferring chat LLMs. For instance, TCOT enables the transfer of models without requiring advanced supervised data, such as human preference data. We enhance TCOT's performance by incorporating pre-training on translation and target language modeling. One of the key innovations in our approach is the Recovery KD technique, which, to the best of our knowledge, has not been proposed in prior work. Recovery KD allows us to preserve the original knowledge with LoRA fine-tuning.

Furthermore, our empirical findings are novel in demonstrating that the commonly used GPT-4 KD ( i.e. Alpaca dataset) is misleading when applied to chat LLM transformation. We demonstrate that Recovery KD combined with LoRA effectively recovers knowledge by enabling the LLM to learn generalizable patterns, which reduces the impact of LoRA parameters when generating English content. This insight has significant implications for mitigating catastrophic forgetting and could be applied to other tasks as well.

While novelty is one aspect of our contribution, our work also addresses a practical and significant challenge, as highlighted in your Strengths section. Our experiments demonstrate that we achieve superior performance despite using limited resources. More importantly, it contributes to enhancing the safety of non-English LLMs.

Multilingual Baselines. As you suggested, Llama 3.1 serves as a solid baseline for multilinguality. In our previous experiments, we compared Llama 3.1 on TH in Table 5, demonstrating that TransLLM improves the original TH performance. In Response to Weaknesses 3-4, we further enhance Llama 3.1's multilingual capabilities with MSFT. However, TransLLM still significantly outperforms MSFT across languages. Generally, multilingual LLMs perform on par with, or even worse than, language-specific models. In our previous experiments, we have shown superior performance against strong baselines tailored to individual languages. To further support this, we compare the Aya model (https://huggingface.co/CohereForAI/aya-101) on the Aya dataset in PT, TR, and TE, as shown in the table below. TransLLM outperforms the Aya model by a substantial margin.

Comparison between multilingual TransLLM and Aya-model on Aya-dataset:

Language Resource Variation. Regarding the diversity of language resources, our experiments indeed cover a broad spectrum of resource availability. Following the categorization used in the Aya dataset (Table 5), we included languages such as Arabic (high resource), Thai, Portuguese, and Turkish (medium resource), and Telugu (low resource).

We hope that these clarifications will address your concerns.

Thanks for your time and constructive comments.

Weaknesses 1: The evaluation benchmark is mainly obtained from translation. The training strategies involved decide that the chat models will mainly rely on their reasoning and generation capabilities in English, this is even further enhanced by training on translation tasks, therefore they can consequently perform well on the translated benchmarks. I wonder if the authors have tested on non-translated benchmarks?

Yes, we had evaluated our method on non-translated benchmarks. Specifically, the human-annotated test set of the Aya-Dataset used in Table 6 serves as a non-translated benchmark. Singh et al. (2024) encourage annotators to contribute fresh samples representative of their native language. We would like to discuss this point to further improve the significance of our method.

For safety, we also engaged professional translators to test a small set of queries in TH and AR (fewer than 10 queries) designed to prompt TransLLM into generating taboo content related to local customs, politics, and similar sensitive topics. In all cases, TransLLM successfully refused to generate harmful content.

Additionally, constructing multilingual benchmarks by translating English datasets is a widely adopted practice due to resource constraints. For example, XCOPA (Ponti et al., 2020) and the dolly-human-edited test set of the Aya-Dataset (Singh et al., 2024) follow this approach. Similarly, we applied this methodology to MT-Bench and Alpaca-Eval, which encompass diverse domains. To ensure high-quality translations, we collaborated with professional translators, aligning with the standards set by prior works.

Weaknesses 2: The evaluation is done by comparing with GPT4/ChatGPT via using GPT4 as a judge despite the authors mentioning that GPT4 and human annotators exhibit high consistency in the rating process. I wonder if there are accuracy measures for evaluation datasets like XCOPA instead of comparison with the GPT4 to more accurately observe the model’s performance. It would also be helpful to provide example generations to observe models’ responses more qualitatively after adaption for more thorough error analysis.

We appreciate your constructive feedback and would like to clarify a few points regarding our evaluation methodology.

In our paper, we evaluated TransLLM across various dimensions, including instruction following, multi-turn conversation, commonsense reasoning, and safety. Specifically for commonsense reasoning and safety, we provided accuracy measures on benchmark datasets such as XCOPA (Table 17) and AdvBenchMark (Table 3).

For multi-turn conversation, labeling outputs as strictly correct or incorrect may fail to capture nuances and subtle differences in conversational quality. Consequently, comparative evaluations between model outputs have become a widely accepted practice in this domain (e.g., Zheng et al., 2024; Üstün et al., 2024). To provide a more discriminative analysis, we included evaluation scores (on a 1-10 scale) judged by GPT-4, as shown in Tables 18-21.

Additionally, we acknowledge the value of qualitative insights and case studies. As mentioned in Section 4.3 of our paper, we analyzed error cases and attributed most of the observed issues to limitations in English capabilities during the TCOT process. We would be happy to include further example generations to enhance qualitative observations and facilitate a more detailed error analysis. We hope these clarifications address your concerns regarding the robustness of our evaluation approach.

Question 1: Since the strategies mentioned in the paper can effectively reduce catastrophic forgetting. Have the authors tried to transforming the chat models to more than one languages at the same time and compare their performances across languages? Also does the model still maintain high performance in English after the transformation?

We tried to transform Llama-3.1-instruct to five langagues at the same time, with the results presented in Tables 5 and 6. TransLLM demonstrates competitive performance across languages. The model still maintains high performance in English after the transformation, as illustrated in the subsequent table. The performance of the multilingual TransLLM in English is comparable to that of the original LLM (Llama-3.1), with no statistically significant difference.

Comparison between multilingual TransLLM and Llama-3.1-instruct on MT-bench in English:

References

Singh et al., 2024. Aya dataset: An open-access collection for multilingual instruction tuning

Ponti et al., 2020. XCOPA: A multilingual dataset for causal commonsense reasoning.

Zheng et al., 2024. Judging llm-as-a-judge with mt-bench and chatbot arena.

Üstün et al., 2024. Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model.

Thanks for your comments again. We would like to know if our response addressed your concerns. We look forward to discussing these concerns with you.

Thank you for providing the updated the results promptly! I'll raise my score to 6 and also contribution and presentation scores.

Thanks for your time and postive comments.

Weakness 1: Include several baselines to decouple the improvement from the new data vs. the new training scheme.

Thank you for your constructive suggestions. We have added these two baselines following the ablation study settings. As shown in the table below, TransLLM significantly outperforms the baselines, demonstrating the effectiveness of the proposed training scheme. While Recovery KD provides only a slight improvement, combining Recovery KD with LoRA helps recover English knowledge. However, without TCOT, the model struggles to fully leverage this English knowledge.

Comparison between TransLLM and baselines on MT-bench when transforming Llama-2 in Thai:

Question 1: For the English training resources, is the translation performed for each target language, with the same amount of translated pairs?

Yes, the same Recovery KD dataset is translated into each target language, ensuring that all translated pairs are parallel and have the same quantity.