Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation

Thanks for your efforts to provide insightful comments. We address your concerns point by point below.

W1a: The novelty of our work.

We would like to clarify that RaDis is a Continual Instruction Tuning (CIT) method and we mainly ground it to translation tasks in our paper. We believe that our paper made several unique contributions and its novelty is also recognized by the other three Reviewers:

First, as demonstrated in General Response 1, RaDis introduces a novel paradigm for building LLMs with both translation proficiency and general abilities. Specifically, previous works (e.g., BayLing, TowerInstruct) have primarily adopted a multi-task training paradigm, which fine-tunes a base LLM on both translation and general instruction data together. In contrast, RaDis follows a continual learning (CL) paradigm, fine-tuning chat LLMs on translation data while leveraging continual learning techniques to mitigate the forgetting of general abilities. Our experiments show the superiority of this paradigm in preserving general abilities, while also demonstrating its competitive performance and future potential in translation tasks.

Second, RaDis is a novel CIT method with a strong motivation. To the best of our knowledge, we are the first to observe that LLMs tend to generate detailed rationales for translation tasks without any explicit chain-of-thought (COT) prompting. Furthermore, we are also the first to leverage this observation by proposing a CIT method that distills self-generated rationales.

Regarding the specific concern you raised, RaDis is a continual learning (CL) method and therefore is not related to prior RLHF works. The only similarity between the two is that both approaches use self-synthesized data. However, the use of the model itself to generate training data is a widely adopted approach in the ML/NLP field, especially in distillation-related methods. Given that the core contribution of RaDis lies in the innovative construction of distillation data, we do not believe this diminishes the novelty of our approach.

In terms of COT distillation methods in the reasoning field, we have included a dedicated section in the Related Work to contextualize our approach. Specifically, prior research in the COT reasoning domain typically involves prompting an external model to generate a reasoning path, which is then used to train the student model to enhance its reasoning capabilities. In contrast, RaDis leverages intrinsic, self-generated rationales as replay data in CL to preserve the model's original abilities. These distinctions make RaDis fundamentally different from prior studies and provide fresh insights into the application of CL for LLMs.

W1b&Q2&Q3: The quality of self-generated rationales. Using reward models for on-policy optimization.

Your suggestion to use a reward model to evaluate the rationale quality is truly insightful. As shown in Table 5, higher-quality rationales can indeed lead to better translation performance. However, we hope to argue that the goal of distilling self-generated rationales is to preserve the original output distribution. From that perspective, the optimal choice is to utilize exactly the same output sampled from the model itself, which shall minimize the distribution gap. As shown in Table 5, Llama-3-70B, a stronger and larger model than Mistral-v0.2-7B, could generate higher-quality rationales. However, fine-tuning with these high-quality rationales does not lead to better general abilities. This is mainly because these rationales are out-of-distribution and would disrupt the output distribution of the student LLM, which has proven to cause forgetting [1].

It is possible that sampling rationales from the output distribution and choosing high-quality ones for training may not cause a huge disruption to the model's output distribution. However, such a reward model needs to be trained for each backbone LLM and requires substantial human annotation. Unfortunately, we do not have the resources to do this at the moment. We leave this investigation to our future work.

W2&Q1&Q4: Comparisons to Tower and multi-task training baseline.

Please refer to General Response 1, where we demonstrate that RaDis outperforms both Tower and multi-task training baseline in general abilities and performs competitively in translation performance.

[1] Yang, Zhaorui, et al. "Self-distillation bridges distribution gap in language model fine-tuning." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). (2024).

Thank you for your response. We would appreciate the opportunity to engage in a more detailed discussion with you.

Fairness regarding utilizing chat LLMs.

The traditional training approach is unable to leverage chat LLMs. Because they suffer from severe forgetting caused by continual pre-training (CPT) or vanilla fine-tuning. In contrast, RaDis allows for the retention of the general abilities of chat LLMs. We believe that utilizing accessible chat LLMs represents a valuable contribution, as it addresses the limitations of previous methods.

Utilizing an RM to filter rationales in RaDis.

We acknowledge that implementing an 'RLHF-based' version of RaDis could potentially improve results. However, the substantial resources required to train an RM were not available to us. Besides, introducing an RM will also raise questions on whether the general abilities are preserved by rationales or learned from the RM.

Additionally, the specific implementation is not the focus of this paper. Our key contribution is introducing rationale in CL, which brings new insights on relying on the reasoning ability of LLMs to connect learned and new knowledge and alleviate forgetting. The filtering technique is orthogonal to our contribution (the rationales are still self-generated whether they are filtered or not) and does not hurt novelty.

Rationale quality.

To further address your concern regarding the rationale quality, we randomly sampled 200 sentence pairs and examined their corresponding rationales. Our analysis found that most of the rationales are highly related to the corresponding sentence pairs.This is because the model can refer to the reference translation when generating rationales, which is consistent with prior work [1] showing that post-rationalization helps reduce hallucinations. We have uploaded the rationale samples as supplementary materials for further reference.

Whether RaDis is just a simplified version of current aligning approach.

We would like to emphasize the fundamental differences between RaDis and RLHF.

The objective of RaDis is to fine-tune the model for a specific task while preserving its original abilities. This involves training with task data. Self-generated data is used to mitigate forgetting.

In contrast, RLHF aims to aligning model behavior with human preferences, with little or no task-specific annotated data. The model learns from human feedbacks. Self-generated data is used as a bridge to convey human preferences.

While both approaches use self-generated data, it serves different purposes, in different ways, with distinct motivations. Therefore, we respectfully disagree with the reviewer’s characterization of RaDis as a simplified version of current aligning approach. As using synthetic data for training is currently one of the most common methods for fine-tuning LLMs, it would not be fair to claim that a paper lacks novelty because it utilizes synthetic data for training. While the training approach may seem similar on the surface, we believe that the reviewer may have overlooked the novel insights introduced by RaDis (utilizing rationales to alleviate forgetting), as well as the differences between our method and the existing post-training framework.

Why RaDis presents non-trival contribution

We believe the reviewer's claim that 'using self-generated data to preserve a chat-LLM's ability is not surprising' oversimplifies the problem and our approach.

First, RaDis involves SFT with task data. In contrast, in RLHF (on-policy), the model is fine-tuned exclusively with self-generated data, with 10-100x smaller learning rate. Given these difference, the findings that forgetting can be easily mitigated with self-generated data in the RLHF phase do not necessarily generalize to the SFT phase.

Second, not all self-generated data is equally effective in alleviating forgetting. As demonstrated in our experiments, SDFT also uses self-generated data during training. However, its effectiveness in mitigating forgetting is clearly inferior to that of RaDis.

Finally, self-generated data may hinder new task learning. As demonstrated in our experiments, Seq-KD and SDFT—two baselines that also utilize self-generated data—failed to improve the model's translation performance, while RaDis balances both learning and consolidating.

The clarifications above demonstrate that achieving effective CIT, which requires both alleviating forgetting and learning new knowledge, with synthetic data is not a trivial problem. Therefore, we sincerely hope the reviewer could reconsider the novelty of our proposed approach. If the reviewer still has concerns, we would sincerely appreciate references to prior works that may address similar ideas.

[1] Chen, Xiao, et al. "Post-Semantic-Thinking: A Robust Strategy to Distill Reasoning Capacity from Large Language Models." arXiv preprint arXiv:2404.09170 (2024).

Dear Reviewer cw7y,

Thank you for the time and effort you've dedicated to reviewing our work. As the discussion phase comes to a close, we kindly ask that you review our follow-up responses (including the new general responses), which aim to address your latest questions.

While we recognize there may still be differences in opinion, we deeply appreciate the thoroughness of your feedback. We understand that providing multiple rounds of detailed commentary is both rare and invaluable. Our responses aim to foster a deeper discussion of differing viewpoints. Regardless of whether you choose to maintain or revise your score, we fully respect your decision.

Thank you again for your time and consideration.

Best regards,

Authors

We appreciate your questions and value your feedback. Below, we provide detailed explanations to address your concerns.

W1a: Applicability in real-world scenarios.

First, we would like to clarify that RaDis is a continual learning (CL) method. A key challenge in CL is known as the stability-plasticity trade-off, where excessive learning plasticity or memory stability can compromise each other. As a result, achieving the same performance as vanilla fine-tuning while preserving all of the model's original abilities is inherently difficult [1,2].

Table m1: Experimental results with Qwen2.5-Instruct.

In our original paper, the average translation performance with Mistral-v0.2 as the backbone LLM is 84.31/82.33 for Vanilla-FT and 84.39/81.94 for RaDis. The average gap between RaDis and Vanilla-FT is only 0.155 COMET22 across eight translation directions. As shown in Table m1, switching the backbone to Qwen2.5-Instruct narrows this gap further to 0.11 COMET22, which is a relatively small performance difference. On the other hand, RaDis consistently outperforms Vanilla-FT in preserving general abilities, retaining 82.31% more of these abilities on average. Given these results, we believe that RaDis is competitive in task-specific performance and superior in general task performance, making it a highly effective fine-tuning method applicable to real-world scenarios.

W1b: Comparison between RaDis and the backbone LLM.

Thank you for your suggestion. In our original paper, we compared RaDis with the backbone LLM primarily to demonstrate that it enhances translation performance while preserving general capabilities. Comparing RaDis with Vanilla-FT highlights that RaDis not only achieves comparable translation performance but also significantly mitigates the forgetting of general abilities. We believe that both comparisons convey the same core message: RaDis is an effective method for continual instruction tuning. Following your suggestion, we have modified the sentence to improve clarity.

W2: Utilizing a stronger teacher model.

We are happy to address the concern regarding the use of a stronger teacher model. As explained in Section 4.2, Seq-KD and SDFT are part of the Continual Instruction Tuning (CIT) baseline. In the continual learning setting, only the dataset for the current task and the model to be trained are available. As we focus on preserving the ability of the original LLM instead of learning from a stronger teacher, most CL approaches that involve knowledge distillation are limited to self-distillation [1]. Therefore, we believe our experimental setup is well justified.

[1] Wang, Liyuan, et al. "A comprehensive survey of continual learning: theory, method and application." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

[2] Wu, Tongtong, et al. "Continual learning for large language models: A survey." arXiv preprint arXiv:2402.01364 (2024).

W3: Clarification on the ablation results

Here we provide further clarification on the results presented in Section 5.2. There appear to be some typos in Lines 462-463: "These results suggest that models can learn stronger general （translation） capabilities through higher-quality rationales, which leads to better overall performance." Our finding is that translation performance can benefit from higher-quality rationales generated by stronger teacher models, which aligns with previous work on chain-of-thought distillation methods in reasoning fields [4].

Regarding the preservation of general abilities, we found that using the model itself as the teacher is the most effective approach. This is primarily because rationales generated by an external teacher model tend to be out-of-distribution relative to the student model. Fine-tuning with these out-of-distribution rationales can significantly corrupt the model’s output distribution, which has been shown to lead to forgetting [3].

Q1: Testing RaDis on broader tasks and comparing it to SDFT.

In our experiments, we observed that paraphrasing translation references generates completely irrelevant sentences when using Mistral-v0.2 as the backbone, which leads to significantly worse translation results.

In General Response 2, we conducted experiments on code generation tasks using the Magicoder dataset. Compared to translation, code generation is more deterministic, and our results show that RaDis still outperforms SDFT in this task. Please refer to General Response 2 for more detailed results and analysis.

Typos.

Thank you for pointing this out. We will correct these typos in a revised version of our paper.

[3] Yang, Zhaorui, et al. "Self-distillation bridges distribution gap in language model fine-tuning." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). (2024).

[4] Wang, Peifeng, et al. "SCOTT: Self-Consistent Chain-of-Thought Distillation." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Dear Reviewer M6wr,

We appreciate your insightful feedback. Following your suggestions, we have carried out further experiments and made revisions to the paper. As we are approaching the end of the discussion phase, we would like to know if our responses have resolved your concerns. We look forward to your response.

Best, Authors

Thanks for your recognition of our work and insightful questions. We believe they hold significant value for our work. We hope the following responses can help address your concern.

W1: Testing RaDis on broader tasks.

Please refer to General Response 2, where we conducted experiments on code generation tasks using the Magicoder dataset. The results demonstrate the superiority of RaDis and highlight its potential.

Given that our current paper primarily focuses on translation tasks, and considering the limited time during the discussion phase, testing RaDis on a broader range of tasks may not be feasible at this stage. However, we greatly appreciate your insightful suggestion and plan to further develop RaDis as a robust, general-purpose continual instruction tuning method in our future work.

W2: Experiment setting in Section 5.2.

We are happy to clarify the rationale behind the specific setting in Section 5.2. The primary goal of this section is to conduct an ablation study to analyze which aspects of RaDis contribute most to its success. As noted in the paper, we focus on two key factors: rationale quality and the self-distillation property. However, since these two factors are generally intertwined in the generated rationales, we needed to design specific experiments to separate them. To do so, we incorporated a stronger teacher from a different model family.

Table j1: Rationale for selecting different teachers.

As shown in Table j1, with Mistral-v0.2 as the backbone model, we can categorize the teacher models along two axes: rationale quality and whether they ensure the self-distillation property. It is straightforward to observe that a larger and more capable teacher model tends to generate higher-quality rationales. However, ensuring that the teacher model does not satisfy the self-distillation property requires more careful design. Since models from the same family often share a significant portion of their training data, we opted to use models from completely different model families. This ensures that the generated rationales are out-of-distribution relative to the student model. By doing so, we can effectively separate the rationale quality and self-distillation property and access their contributions, respectively.

Q1: Ablation experiments on the percentage of examples with rationales

Table j2: Ablation experiments on the percentage of examples with rationale.

Your suggestion to include an ablation study on rationales has been constructive. We conducted experiments with varying percentages of examples containing rationales (25%, 50%, 75%) and would like to share the results.

First, the results show that RaDis achieves the best overall performance on general tasks when rationales are provided for the entire training dataset. However, adding rationales to just 25% of the training data significantly mitigates forgetting on general tasks. Interestingly, the effectiveness of rationales appears to vary across task types: instruction-following and safety tasks are the easiest to preserve, while math reasoning tasks are the most challenging.

Moreover, scaling the percentage of data with rationales from 25% to 75% does not result in a linear improvement. Notably, the model's performance changes dramatically in two key intervals: from 0% to 25% and from 75% to 100%. These findings highlight that the inclusion of rationales has a non-linear impact on performance, with significant gains observed at the extremes.

We believe this phenomenon could be explained as follows:

1. Superficial Alignment Hypothesis: As defined in LIMA [1], the Superficial Alignment Hypothesis suggests that most of a language model’s abilities and knowledge are acquired during pre-training, while post-training primarily focuses on refining the model's style and format. Building on this hypothesis, we interpret our ablation results to indicate that the forgetting of superficial alignment can also be mitigated superficially. For instance, incorporating a small number of rationales (25%) is sufficient to recover the model’s performance on instruction-following and safety tasks. However, this level of rationale integration is adequate for more complex skills, such as math reasoning, which does not obey the Superficial Alignment Hypothesis.

2. Distribution Gap of Training Data: Traditionally, when training with mixed data from two tasks, the results are expected to be a linear combination of training each task independently. However, due to the vast number of parameters in LLMs, these models can effectively memorize different tasks in separate parameter spaces based on their input patterns. This phenomenon has also been observed in the Llama3 technical report, where researchers from Meta utilized distinct system prompts to toggle the model between different modes (e.g., text or speech).

   As a result, the mixup ratio of training data may not be a critical parameter. Instead, the primary factors are the tasks included in the training set and whether they are in-domain or out-of-domain. When 0% of the data contains rationales, the model is trained solely on translation tasks (out-of-domain), leading to severe forgetting of general abilities. By incorporating a mix of translation and rationale-augmented translation (which can be considered a form of instruction-following translation task), the model learns both tasks simultaneously. However, the existence of an out-of-distribution task still leads to forgetting. When the entire training set includes rationales, the model is trained predominantly on in-domain tasks, significantly alleviating the forgetting of general abilities.

We hope the experimental results and analysis presented above adequately address your question. If you have any further concerns, we would be delighted to engage in a more in-depth discussion with you.

[1] Zhou, Chunting, et al. "Lima: Less is more for alignment." Advances in Neural Information Processing Systems 36 (2024).

Thank you for your response! We're glad to hear that many of your issues have been resolved. We are conducting experiments on additional tasks and will provide the results as soon as possible. Besides, we've provided a new general response, explaining why we would view CL and MTL two different learning paradigm and why investigating CL for LLM is crucial and irreplaceable in the application of LLMs.

Dear Reviewer JdnZ,

Thank you again for the time and effort you’ve dedicated to reviewing our work. As the discussion phase is coming to a close, we kindly ask if you could review our follow-up responses. We’re grateful that you’ve mentioned most of your initial concerns have been addressed by prior rebuttal, and we have also provided replies to your follow-up requests for testing our approaches with broader tasks.

If there are no further concerns, we would be grateful if you could reconsider the score.

Thank you for your time.

Best regards,

Authors

Thank you once again for your feedback! We have conducted additional experiments on TriviaQA (open-ended QA) and OpenFunctions (function call). Below are the results from these experiments:

As demonstrated in the tables above, RaDis effectively alleviates forgetting and outperforms SDFT on both tasks. Additionally, it achieves superior performance on the fine-tuned task compared to Vanilla-FT. These results support our claim that rationales play a crucial role in bridging learned and new knowledge, thereby enhancing both task learning and knowledge retention.

Thanks for your insightful questions and they hold significant value for our work. We appreciate your recognition of the meaning of our research topic—building LLMs that excel in both translation proficiency and general ability. We hope the following discussion will address your concern and show the advantages and potential of our approach.

W1a: The performance seems to be worse on both translation and general ability.

We would like to argue that this comparison is not entirely fair, as it is comparing RaDis with two models in their respective advantage zones. As we have shown in the paper and the general response. RaDis significantly outperforms LLM-based MT models like TowerInstruct in general abilities with 100x less training budget, and it also outperforms its original backbone in translation. Taken together, RaDis is the best model among those tested, successfully achieving both high translation proficiency and robust general abilities.

Given the focus of our paper on building LLMs that excel in both translation proficiency and general ability, we believe it is more comprehensive to consider both aspects in the evaluation. As we have shown, RaDis is indeed the best model from this perspective, achieving strong performance in both areas.

W1b: The dependency on backbone LLM limits the multilingual translation performance.

Regarding your concern that the translation performance of RaDis is limited by the choice of backbone LLM, we believe this limitation can be effectively addressed. As demonstrated in General Response 1, switching the backbone from Mistral to Qwen2.5 leads to substantial improvements in translation performance, effectively narrowing the gap caused by multilingual training. Moreover, with the emergence of open-source multilingual LLMs (e.g., Aya, Llama-3.1, Qwen2.5), the impact of multilingual training may diminish over time. As stronger backbone LLMs continue to develop, RaDis's translation proficiency can be further enhanced, while its superiority in general tasks will remain intact.

Overall, from the perspective of building an "almighty" LLM, RaDis introduces a novel continual learning paradigm that is not only competitive but, in many respects, superior to the traditional multi-task training approach. It demonstrates significant advantages in preserving general abilities, while also showing strong potential in translation tasks.

Thank you for your response. I sincerely appreciate the opportunity to engage in a more in-depth discussion with you.

The 'Awkward Position'

As we stated in Lines 042–043, our goal is to develop models that seamlessly integrate strong translation performance with broader general-purpose utility. And we successfully achieved it by enhancing the translation performance of instruction-tuned LLMs while preserving most of their general abilities. It is important to clarify that "strong" does not imply "state-of-the-art" (SOTA), and at no point have we claimed achieving SOTA performance as either a goal or a contribution of our work. (Frankly, we are surprised that we need to debate about what we wrote in the introduction and the interpretation of this specific word.)

In response to your initial review regrading the gap in translation performance with SOTA LLM-based MT models such as ALMA, we conducted an additional experiment, as presented in General Response 1. The results demonstrate that RaDis, using Qwen-2.5 as its backbone, achieves strong translation performance that surpasses ALMA on the WMT23 test set. Simultaneously, RaDis significantly outperforms ALMA and TowerInstruct in terms of general-purpose capabilities. A 27.91 win rate on Alpaca Eval 2 leaderboard [1] means our model is better than Gemini Pro and Claude 2.1, which are strong chat LLMs. Based on these findings, we believe our claims are well-supported: a model with translation performance surpassing ALMA and general ability outperforming Gemini Pro is not awkward; in fact, it is quite impressive.

Considering the stated goal of our paper, we believe a more appropriate question to evaluate our work would be: If an end user seeks a versatile model that is good at translation, provides strong chat capabilities, and can assist with a wide range of other potential tasks, would they choose ALMA, Llama-2-Chat, or RaDis? We feel this question better aligns with the intended scope and contributions of our work. In practice, such a model could indeed achieve better interactivaty. With strong instruction-following capabilities, RaDis can adapt translations to a specified style, incorporate specific terms as requested, or even provide explanations for translated sentences. These features are beyond the capabilities of traditional LLM-based MT models.

While it is always easy to request SOTA performance on individual tasks, building a model that achieves SOTA across every task simultaneously is highly non-trivial. Although RaDis does not achieve SOTA performance in every task, it successfully enhances the translation performance of instruction-tuned LLMs, creating a model that is "strong" across various tasks—a balance that aligns with the overarching goal of our work and meets the need for general-purpose LLM.

P.S. After reading the response from the reviewer, we still find the phrase "awkward position" to be unclear. In this response, we have tried to address this concern by elaborating on both the performance and practical advantages of our approach. We kindly request that the reviewer provide a more specific criterion or clarify what defines an "awkward" versus "not awkward" position. This would greatly help us better understand and address your concerns.

Translation performance on low resource languages.

First, we would like to clarify that the experiment in Table g1-1 was conducted under exactly the same settings described in our paper, using Czech, German, Russian, and Chinese. For Table g1-2, we followed the experimental setup outlined in the TowerInstruct paper and directly referenced the results reported there. The evaluation in Table g1-2 was performed on German, Russian, and Chinese, as TowerInstruct was trained on the following 10 languages: English (en), German (de), French (fr), Dutch (nl), Spanish (es), Portuguese (pt), Russian (ru), Chinese (zh), Korean (ko).

Given that RaDis is a post-training method, its performance is inherently influenced by the language proficiency of the backbone LLM. As a result, achieving improved translation performance on low-resource languages not included in the backbone LLM's training set is particularly challenging.

However, we want to emphasize that the goal of our experiment with Qwen-2.5 was to demonstrate that, when equipped with a stronger multilingual backbone, the performance gap caused by additional multilingual pre-training can indeed be reduced. This is an important finding because multilingual backbone LLMs are continually evolving and becoming more capable. As such, we believe our method holds strong potential for future advancements.

Inference speed.

We conducted inference experiments using vLLM [2]. This framework includes a 'stop/stop_token_ids' parameter, which allows the generation process to halt when specific strings or tokens are produced.

[1] https://tatsu-lab.github.io/alpaca_eval/

[2] https://docs.vllm.ai/en/latest/dev/sampling_params.html

Learning paradigms and the accessibility of training data.

First, it is generally acknowledged that continual learning is recognized as a distinct learning paradigm [3,4]. We used this term to align with established terminology in prior works within this field.

Second, while the reviewer claims that task-specific data is always accessible, this is not the case. As we stated in Lines 047-049, the training data for open-sourced chat LLMs is frequently unavailable, making multi-task training with the original data infeasible. This limitation was a primary motivation for us to explore the continual learning paradigm.

Besides, our experiments with open-sourced instruction data clearly demonstrate that relying solely on these datasets is insufficient to effectively mitigate the forgetting of general capabilities. In contrast, our approach using continual learning largely preserves the general capabilities of chat LLMs and outperforms the multi-task training baseline requested, which highlights a significant advantage of our method.

Finally, we believe it is important to note that, although our evaluation focused on instruction following, safety, and math reasoning, general ability encompasses a wide range of tasks (e.g., code generation, semantic classification, summarization, etc.). While it is possible to incorporate additional task-specific datasets into training to improve performance on individual tasks, this approach comes with significant costs, including the effort required to collect such datasets and the computational expense of fine-tuning to achieve balanced performance across tasks.

In contrast, our method avoids these unnecessary training costs by leveraging continual learning to preserve the knowledge embedded in backbone LLMs. This aligns with one of the key motivations behind the continual learning paradigm: to retain knowledge in pre-trained models while integrating new information.

From a broader perspective, RaDis has the potential to serve as a general-purpose continual instruction tuning method, a point that was notably recognized by Reviewers JdnZ and M6wr.

We hope this clarification highlights the strengths and potential of our approach and may encourage you to reconsider our contribution in a more positive light. Thank you again for your valuable insights and constructive feedback—they have been instrumental in helping us better articulate our work.

[3] Liu, Bing. "Lifelong machine learning: a paradigm for continuous learning." Frontiers of Computer Science 11.3 (2017): 359-361.

[4] Wang, Liyuan, et al. "A comprehensive survey of continual learning: theory, method and application." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

Thank you for getting back to me so quickly. We are glad to see that you have raised the score. We would like to take this opportunity to provide additional information to address your remaining concerns.

Dependency on backbone LLM.

There still seems to be some misunderstanding. As we stated in our previous response, RaDis is a post-training method, and it is generally acknowledged that the language proficiency of the pre-trained LLM influences post-training methods. In this context, dependency on the backbone LLM primarily affects its language proficiency.

Robustness against scaling model size.

Regarding the new concern raised by the reviewer—robustness against scaling model size—we refer to Table g1-2 in General Response 1, where we have demonstrated that switching to a stronger backbone (Qwen2.5) leads to improvements across all tested tasks. Additionally, we have implemented RaDis on three different LLMs—Llama-2-Chat, Mistral-v0.2-Instruct, and Qwen2.5-Instruct—and across two different tasks (machine translation and code generation). The performance of RaDis remains highly consistent in these settings. These results clearly indicate that RaDis is a robust approach that generalizes well to different backbone models and tasks. Based on this evidence, we expect similar robustness against scaling model size.

We are happy to provide additional experimental results on scaling model size if the reviewer insists.

Whether the performance is impressive.

As we have reached an understanding that achieving SOTA performance is not a requirement for this work, we would like to highlight the impressive performance of our method from the perspective of LLM-based MT models, which appears to be a major focus of the reviewer.

As shown in Table g1-2, RaDis consistently outperforms TowerInstruct—a SOTA LLM-based MT model with general instruction-following capabilities—on general tasks, regardless of the backbone utilized. Specifically, RaDis demonstrates a +34% win rate on AlpacaEval, +69% safety improvement on AdvBench, and +81% accuracy improvement on GSM8K. While achieving this superior performance on general tasks, RaDis also delivers strong translation performance, outperforming ALMA. Additionally, all these results are achieved with only 1% of the training cost compared to TowerInstruct.

Taken together, RaDis provides an impressive LLM-based MT model that delivers SOTA performance on general instructions (when focusing on LLM-based MT models specifically) while maintaining strong translation capabilities, all while utilizing over 100x less training compute.

The novelty and contribution of our work.

We respectfully disagree with the reviewer's claim that our approach is engineering-focused and would like to emphasize that our paper made several unique contributions:

First, as demonstrated in General Response 1, RaDis introduces a novel approach for building LLMs that combine translation proficiency with general abilities. Previous works (e.g., BayLing, TowerInstruct) primarily rely on a multi-task training paradigm, fine-tuning a base LLM on both translation and general instruction data simultaneously. In contrast, RaDis adopts a continual learning (CL) paradigm, fine-tuning chat LLMs on translation data while employing continual learning techniques to mitigate the forgetting of general abilities. Our experiments highlight the superiority of this approach in terms of broad general abilities and training efficiency, while also showcasing its competitive performance and promising potential in translation tasks.

Second, RaDis introduces a novel Continual Instruction Tuning (CIT) method with a strong motivation. To the best of our knowledge, we are the first to observe that LLMs tend to generate detailed post-rationales even without explicit chain-of-thought (CoT) prompting. Furthermore, we are the first to leverage this observation by proposing a CIT method that distills self-generated rationales. Our experiments demonstrate that RaDis is a robust CIT approach, capable of generalizing effectively across different backbone models and tasks. We believe introducing rationales to CL brings new insights into relying on the reasoning ability of LLMs to connect learned and new knowledge and alleviate forgetting.

Given these points, we believe your concerns regarding the novelty of our work can be effectively addressed. Nonetheless, we will further emphasize our contributions in the revised version of the paper. If there are specific concerns regarding our contributions—such as previous works that may have addressed similar ideas—we kindly ask the reviewer to provide those references so that we can better understand and address these concerns.

Dear Reviewer pUWZ,

Thank you once again for the time and effort you've dedicated to reviewing our work. As the discussion phase comes to a close, we kindly ask that you review our follow-up responses (including the new general responses), which may address the questions raised.

While we acknowledge any differences in opinion, we deeply appreciate the thoughtful feedback you’ve provided. We understand that multiple rounds of detailed review are rare and invaluable, and our goal with these responses is to foster a more thorough discussion of differing viewpoints. Regardless of whether you choose to maintain or revise your score, we fully respect your decision.

Best regards,

Authors

We have conducted additional experiments on TriviaQA (open-ended QA) and OpenFunctions (function call). Below are the results from these experiments:

As demonstrated in the tables above, RaDis effectively alleviates forgetting and outperforms SDFT on both tasks. Additionally, it achieves superior performance on the fine-tuned task compared to Vanilla-FT. These results support our claim that rationales play a crucial role in bridging learned and new knowledge, thereby enhancing both task learning and knowledge retention.

Table g1-2: Comparison to TowerInstruct-v0.2. The best result in each column is marked in bold. The second best is italicized.

Given that TowerInstruct has been fine-tuned on the WMT22 test set, we shifted the translation test set to the WMT23 test set. We report the performance of the best model in our paper (Mistral+RaDis) alongside TowerInstruct. To further demonstrate the potential of our approach, we also conducted new experiments with Qwen2.5-Instruct as the backbone (Qwen2.5+RaDis).

As shown in Table g1-2, RaDis consistently outperforms TowerInstruct-v0.2 in terms of preserving general abilities. This is primarily because TowerInstruct-v0.2 is fine-tuned using UltraChat, which, like other open-sourced instruction datasets, suffers from lower quality.

In terms of translation, TowerInstruct-v0.2 achieves higher performance, largely due to the benefits of multilingual pre-training and extensive parallel fine-tuning. However, we would like to emphasize the strong potential of our approach from two key perspectives:

1. RaDis is more efficient: The training times for TowerBase 7B and 13B were 80 and 160 GPU days, respectively, using A100-80GB GPUs. Fine-tuning TowerInstruct adds an additional 200 GPU hours. In contrast, RaDis requires only 20 GPU hours (4 hours for generating rationales and 16 hours for training), which is less than 1% of the training cost for TowerInstruct-7B, while still achieving strong performance.
2. RaDis can benefit from stronger backbone LLM: While TowerInstruct achieves better translation performance, RaDis can effectively bridge this gap by leveraging a stronger backbone LLM. As shown in Table g1-2, switching the backbone from Mistral to Qwen2.5 leads to substantial improvements across all tasks and outperforms ALMA. We believe that as open-source multilingual LLMs continue to improve, the performance gap in translation will gradually narrow.

Together, these results underscore the advantages of our approach and demonstrate that RaDis offers a novel and competitive paradigm for building LLMs that excel in both translation proficiency and general ability.

Following the reviewers' suggestion, we will include the additional experiments and discussions comparing RaDis with TowerInstruct and the multi-task training baseline in the revised version of our paper.