Dear reviewer 9qib, thanks for your valuable suggestions. Here are our responses:

Response 1 (Fixed EMA): To address your concern, we have conducted ablation studies with more fixed EMA weights, i.e. 0.993, 0.996 0.999 (as the EWA weight is usually set in [0.990, 0.999]) [1]. The results are shown as:

From the Table, our proposed dynamic EMA method consistently achieves the best performance across all three metrics compared to certain fixed EMA weights (e.g., from 0.990 to 0.996). When compared to a fixed EWA weight of 0.999, our method demonstrates superior Avg.ACC and New.ACC but exhibits inferior resistance to forgetting. It is important to clarify that, as described by Equation (1) in our manuscript, a higher EMA weight generally results in lower Forgetting but also lower New.ACC. However, the lower Forgetting does not always indicate better overall performance due to the trade-off between plasticity and stability. In extreme cases, the Avg.ACC metric may be significantly impacted by a reduced New.ACC. As shown in the Table, under the 0.999 EMA weight setting, the New.ACC metric only reaches 49.44, which is considerably lower than the New.ACC values of other methods (others all exceeding 60). In summary, the low Forgetting in the 0.999 EMA weight setting comes at the cost of significantly reduced New.ACC.

Response 2 (QWen Baselines): Please kindly refer to the Response 1 for the same question of Reviewer 4GQq.

Response 3 (Table Titles): We recognize that some table titles, including Table 6, may be too brief. To improve clarity, we have revised the titles of these tables to include more detailed explanations. For example, we have updated the title of Table 6 to “Ablation study results for each proposed components”.

Response 4 (L1-Norm): We have replaced L1-Norm in our method with L2-Norm and reconducted the whole experiment based on the LLaVA-7B architecture with the Origin instruction type. The results are shown as:

We can see that the L1-Norm consistently outperforms L2-Norm, regardless of Avg.ACC, Forgetting, or New.ACC metrics, which further enhances our motivations of adopting L1-Norm approximation.

Response 5 (Phrase): We have replaced "baseline" with "LoRA Fine-tuning baseline" to specify which baseline is being referred to, making the context clearer for readers. The updated sentence likes: "especially significant enhancements in the performance of the LoRA Fine-tuning baseline (as shown in Figure 1)."

Response 6 (Avg.ACC Results For Instruction Template): As you have pointed out, this phenomenon is closely related to the type of instruction template. As we presented in the Table 9, the 10Type instruction type owns 10 different instruction templates for each task, which require the MLLM to memorize a larger amount of information. Consequently, the challenge of mitigating forgetting is harder compared to the Origin and Diverse instruction types, which utilize only a single instruction template for each task. This conclusion is further supported by the Forgetting values reported in Table 5 (Origin: 1.93, Diverse: 0.45, 10Type: 2.86). Therefore, due to the increased forgetting, the Avg.ACC of the 10Type instruction type is slightly lower than those observed of the Origin and Diverse instruction types.

Response 7 (Forgetting Metric): In fact, both the Forgetting and BWT metrics can be used to measure catastrophic forgetting. The key distinction lies in their typical applications: the BWT metric is more commonly employed in Task Incremental Learning (TIL), whereas the Forgetting metric is often used in Class Incremental Learning (CIL) [1,2]. In our paper, the choice of evaluation metric is guided by the need to ensure consistency with the benchmarks utilized [3,4]. Therefore, we follow their original metric setting in our evaluation.

[1] Trgp: Trust region gradient projection for continual learning.

[2] Learning to prompt for continual learning.

[3] Coin: A benchmark of continual instruction tuning for multimodel large language model.

[4] Citb: A benchmark for continual instruction tuning.

We would like to express our sincere thanks for your thorough review and valuable suggestions. Your thoughtful comments have been crucial in identifying areas for improvement, allowing us to refine and enhance the quality of our paper. Thank you again for your effort and dedication in helping us improve our work.

Dear reviewer 4GQq, thanks for your valuable suggestions. Here are our responses:

Response 1 (QWen Baselines): We newly implemented another two strong baselines: EProj and EWC on the QWen-VL architecture. The results are presented as:

The results demonstrate that our method still owns the best performance, surpassing other state-of-the-art approaches.

Response 2 (Resource Consumption): Please kindly refer to the Response 1 for the same question of Reviewer pe71.

Response 3 (Phrase): In the revised version, we have modified the sentence in lines 412-414 as follows: "Additionally, we are surprised to find that our method can spontaneously suppress the occurrence of hallucinations in continual instruction tuning." We also conducted a thorough revision of the paper to identify and eliminate other redundant expressions, ensuring the writing remains clear and precise.

Response 4 (Terminology): We have revised lines 90-92 as follows: "Our method is model-agnostic and can be easily applied to a wide range of CIT methods." Additionally, we carefully reviewed the manuscript to ensure consistent and accurate use of technical terminology throughout the paper.

Response 5 (Expression): To address this, we have clarified the distinction between the "forgetting metric" and "catastrophic forgetting" by adding the explanation that, in our paper, "Forgetting" (with an uppercase "F") refers to the "forgetting metric," whereas "forgetting" (with a lowercase "f") denotes the "catastrophic forgetting.”

Response 6 (Instruction Grouping Results on LLM): To address your concern, we have presented the instruction grouping results on LLM as follows.

Group 1

pubmedqa_classification = ... Output '1' if the passage has a defininte objective/aim/goal and output '0' if the passage does not have a definite objective/aim/goal …

dstc3_classification = ... output the price range the user if looking for which can take one of four values: Cheap, Moderate, Expensive and Don't Care …

air_dialogue_classification = ... select the goal of the conversation …

deal_or_no_dialog_classification = ... answer 'Yes' if both participants agree to the deal, otherwise answer 'No' …

craigslist_bargains_classification = ... classify the text into one of the labels from the two possible outputs - 'accepted'/'rejected' …

Group 2

mutual_multi_turn_dialogue = ... choose the most reasonable option …

personachat_choose_next = ... Choose one and answer with the text …

Group 3

circa_answer_generation = ... generate an answer that is relevant to the question …

convai3_sentence_generation = ... read the input, then generate a valid prediction of the user's response to the computer's clarifying question …

air_dialogue_sentence_generation = ... find the answer of the previous dialogue …

diplomacy_text_generation = ... generate the next message …

curiosity_dialogs_answer_generation = ... find the dialogue that is basically a response given to a question or an aspect of the user ...

smcalflow_sentence_generation = ... identify what will be users' command for that reply ...

multi_woz_user_utterance_generation = ... Generate a language query such that it leads to this reply ...

personachat_generate_next = ... generate the next utterance in a given dialogue ...

dstc3_answer_generation = ... answer the given question based on the information present in the dialogue ...

convai3_sentence_generation = ... generate a prediction of what the requester is actually trying to do ...

smcalflow_sentence_generation = ... return what will be Agent's response/reply for that particular user's command or question ...

Group 4

storycommonsense_motiv_text_generation = ... write the character's motivation by doing a specific job, which is given in the sentence …

It can be seen that instructions for 19 tasks are categorized into 4 groups, which reflects the limited model expansion property of our method.

Thanks for your detailed and constructive review. Your review has significantly contributed to the refinement of our work, offering new insights and helping us address important aspects that were previously overlooked. We have made extensive revisions based on your suggestions, resulting in a clearer and more robust paper.

Dear reviewer pe71, thanks for your valuable suggestions. Here are our responses:

Response 1 (Resource Consumption): We compare the time consumption between the dynamic EMA update and the fixed EMA method under same experimental settings (adopting LLaVA-7B). We measure the time consumption for each task on 4-NVIDIA H100 GPUs. The results are shown as:

We observe that our dynamic EMA update method requires a total of 492 minutes for training across eight tasks. In comparison, the fixed EMA method consumes 455 minutes, indicating that our method only incurs an 8% increase in training time (37 minutes). However, as demonstrated in our experiments, the dynamic EMA update method significantly outperforms the fixed EMA method.

Additionally, we compare the memory consumption of our method with that of approaches that do not use the instruction grouping strategy. Specifically, our method introduces two additional components: one for instruction groups and another for the corresponding LoRA parameters of different instruction groups. Compared to the LoRA parameters, which occupy approximately 1 GB of memory, the saving load for instruction groups—comprising small sentences of natural language text—is negligible, due to requiring only around 1 KB. For simplicity, we define a complete set of LoRA parameters inserted into the LLM as a unit of 1. Under this definition, the method that does not use the instruction grouping strategy shares a single set of LoRA across all tasks, resulting in a quantified memory saving factor of ×1. Our method, using Origin Instruction Type as an example, extends the LoRA parameters across four groups, leading to a quantified memory saving factor of ×4. Given that LoRA is a type of Parameter-Efficient Fine-Tunings (PEFTs), even with a fourfold increase in storage, the total memory load remains smaller compared to the substantial storage demands of LLMs, which often exceed tens of GB [1].

In summary, while adopting the dynamic EMA update and instruction grouping strategy introduces additional time and memory consumption, these costs remain manageable and are significantly outweighed by the substantial performance improvements they promote.

[1] Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Response 2 (Fixed EMA): Please kindly refer to the Response 1 for the same question of Reviewer 9qib.

Response 3 (Instruction Grouping Results on LLM): Please kindly refer to the Response 6 for the same question of Reviewer 4GQq.

Response 4 (Reference): We have incorporated this reference into the Related Work section and discussed it as “After that, TRACE, another continual instruction tuning benchmark is designed to evaluate the general ability, instruction following and safety for LLMs”.

Response 5 (Terminology): We have added clearer explanation and significance of the “plasticity-stability balance” in the Introduction section as “the trade-off dilemma for models to balance the new task learning and old task storing, which is the core of continual instruction tuning”. Besides that, we have checked the whole manuscript to revise other terminologies that have not been explained well.

Response 6 (Expression): We have changed "knowledge confusion" in line 31 to "knowledge interference", which is a more precise and widely accepted term. Besides that, we have checked the whole manuscript to revise other unsuitable phrases.

Response 7 (Dynamic EMA): We have summarized the values of EMA weights at each iteration on TextQA dataset with LLaVA-7B backbone and Origin instruction type. Due to the rebuttal space limitation, we provide the dynamic EMA weight in the first 10 epochs for the trainable parameters in the first three layers.

As we can see that the dynamic EMA weight can really exceed the range of 0-1, which demonstrates our theoretical deduction.

Thank you for taking the time to review our manuscript and provide such detailed and constructive suggestion. Your insights have significantly improved our paper, making it much stronger overall. We have carefully considered all your suggestions and made the necessary revisions to ensure the research is more precise and accessible. We deeply appreciate your contribution to enhancing the quality of our work.

Dear reviewer cghr, thanks for your valuable suggestions. Here are our responses:

Response 1 (Additional Experiments): To address your concerns, we set two diverse task settings: Conventional Image Continual Classification and NLP Continual Classification to further validate the effectiveness of our method.

(1). We choose two Prompt Continual Learning (PCL) methods based on ViT backbone, namely L2P [1] and DualPrompt [2]. The Image Continual Classification Benchmark is dividing a whole image classification dataset into several splits and each split is seen as a task. We freeze the ViT backbone and only train the prompts. Based on the original L2P and DualPrompt baselines, we integrate our dynamical EMA update method. The continual learning dataset is 10-Split-CIFAR100 with Class Incremental Learning (CIL) setting. The experimental results are shown as:

In conclusion, our method exhibits superior Avg. ACC and enhanced anti-forgetting performance compared to the original L2P and DualPrompt.

(2). We implement our method on the NLP Continual Classification Benchmark (MRCC->SST-2->Sentiment140) with Task Incremental Learning (TIL) setting, based on the LLM (Mistral 7B [3]). The NLP Continual Classification is selecting multiple NLP classification datasets and each dataset is seen as a task. We freeze the LLM and insert trainable LoRA paradigm into each layer of the LLM respectively. For baseline comparisons, we adopt the original LoRA and one state-of-the-art continual learning method, CurLoRA [4]. The experimental results are shown as:

It can be seen that our method exhibits comparable stability while achieving superior plasticity in LLM continual tuning tasks compared to the state-of-the-art method in [4]. Notably, both CurLoRA and our method attain zero forgetting due to the simple TIL setting (knowing the task identifier in the testing stage).

The above experiments demonstrate that our method can perform well across diverse task settings and greatly improve the baselines’ performance. These findings further indicate that our method has strong generalization capabilities.

[1] Learning to prompt for continual learning.

[2] Dualprompt: Complementary prompting for rehearsal-free continual learning.

[3] Mistral 7b.

[4] Curlora: Stable llm continual fine-tuning and catastrophic forgetting mitigation.

Response 2 (Impact Statement): Thanks for your kindly reminding. We apologize for the absence of the impact statement. In the revised version, we have added an Impact Statement section to discuss how our approach contributes to continual instruction tuning in LFMs. Specifically, we emphasize that:

1. This paper presents work whose goal is to advance the field of continual instruction tuning and mitigate catastrophic forgetting in LFMs.

2. Its relevance to real-world applications, such as lifelong AI assistants and MLLM continual evolution.

We sincerely hope this addition can provide a clearer perspective on the significance and practical implications of our work.

Response 3 (Multi-Task): (1). The multi-task results presented in Table 1 of our paper are directly from the original CoIN paper (Table 2 in [5]). To maintain the authority of the experimental results, we choose to cite the multi-task results from the original report as a reference. (2). The authors of CoIN found that the performance of multi-task is not always higher than that of LoRA continual learning (shown in the following Table, copied from [5]). They explain that “unlike traditional continual learning, where the multi-task model often serves as the upper bound, in CoIN, the performance of the multi-task model is not the best due to the influence of task gaps”. Notably, this phenomenon is consistent with our results. (3). Our idea is that the theoretical upper bound could be understood as the ideal conditions with zero-forgetting, which is represented by the New.ACC metric (The result of training on the single task). While how to choose the upper-bound in continual instruction tuning is still a valuable research topic.

[5] Coin: A benchmark of continual instruction tuning for multimodel large language model.

We sincerely appreciate your careful review of our paper. Your valuable suggestions have greatly enhanced the quality of our manuscript by pointing out areas that required improvement. We are truly grateful for the time and effort you dedicated to improving our work.