Thank you for recognizing the strengths of our paper and for your effort in reviewing our submission. Below is a detailed response to your concerns:

The Abstract Should Be Shortened

Thank you for the suggestion. We will shorten the abstract in the revised version to improve conciseness.

Difference Between RegLoRA and Existing Regularization-Based Methods

RegLoRA can be viewed as a regularization-based method. However, it differs from most existing approaches in several key aspects:

• As LoRA fine-tuning becomes increasingly prevalent for adapting large-scale models, there is a growing need to mitigate catastrophic forgetting in LoRA-based settings. In this context, RegLoRA—which applies regularization to LoRA adapters—is naturally well-suited, whereas existing regularization methods that operate on the full model may be less appropriate.
• Unlike some existing regularization-based approaches, RegLoRA does not require auxiliary models or a significant number of additional parameters on GPUs. This substantially reduces GPU memory usage.
• RegLoRA focuses on regularizing a small number of key elements rather than entire weight matrices or feature maps, which also distinguishes it from most existing methods in terms of strategy.

In summary, while RegLoRA belongs to the family of regularization-based methods, its design choices make it more efficient and practical for our task setting.

Rationale for Converting Tasks in Five Ways

Our Answer Style Diversification (ASD) paradigm reformulates tasks into five question types based on prior knowledge. Specifically, we examined 15 widely-used MLLM benchmarks and analyzed the applications of MLLM in real-world scenarios. Our analysis reveals that the identified categories—yes/no questions, multiple-choice questions, short answer questions, brief explanation/description questions, and detailed explanation/description questions—cover the majority of use cases. The list of benchmarks referenced is provided in lines L97–L101, all of which belong to these five types. In practical settings, if additional question types arise in specific domains, our approach can also be extended to accommodate them.

Reason for Partially Converting Questions

If all questions are converted, the model cannot learn the answer format of the original question style. Since the test set retains the original format, a model that has never seen samples in this style is unlikely to perform well during evaluation. As shown in Table 3, when the conversion ratio is high (e.g., 80%), performance drops compared to lower ratios (e.g., 20%), indicating that preserving a substantial portion of original-format questions is essential.

Possibility to Combining RegLoRAs and Pseudo-Code Explanation

We agree that combining RegLoRAs—or more generally, LoRAs—from different tasks is an intriguing and promising direction. If successful, this approach could enable the construction of a unified multi-faceted MLLM by integrating task-specific LoRAs, which holds significant practical value. However, this idea lies beyond the current scope of our work. While we are not yet able to evaluate its feasibility or propose a concrete method, we appreciate your insightful suggestion and will continue exploring this direction in future research.

Regarding the overall training procedure of RegLoRA, we provide the pseudo-code below for clarity:

---

Require: base MLLM $M_0$, training sets of all tasks {$D_1, D_2,...,D_T$}
for task $j$ in $1$ to $T$ do
    Insert a new LoRA adapter $LoRA_j$ into model $M_{j-1}$
    for each batch in $D_j$ do
        Compute language modeling loss $L_{lm}$
        if j > 1 then
            Compute regularization loss $L_{reg}$ using all previous masks {$R_1, R_2,...,R_{j-1}$} according to Eq. 2.
            Set total loss $L_{total} = L_{lm} + L_{reg}$
        else
            Set total loss $L_{total} = L_{lm}$
        Update model parameters to minimize $L_{total}$
    Compute the regularization mask $R_j$ for the current task
    Merge $LoRA_j$ into $M_{j-1}$ to form $M_j$

---

Here, the language modeling loss $L_{lm}$ refers to the next-token prediction loss used by the base MLLM.
After learning each task $j$, we evaluate performance on all learned tasks using the updated model $M_j$.

Thank you for your thorough review and for recognizing the value of our contributions. Below, we address your comments in detail:

Training Stages of MLLMs

Thank you for pointing this out. Indeed, the training process of MLLMs/LLMs often involves more than two stages. We will revise the sentence to: "Training these models typically involves multiple phases, with pretraining and instruction tuning being two crucial ones."

Introduction of LoRA Variants

Thank you for your constructive feedback. We will include an overview of LoRA variants, such as those you mentioned: AdaLoRA, which adaptively allocates the parameter budget across weight matrices based on their importance scores; LoRA+, which assigns different learning rates to matrices A and B to enhance performance in settings with large embedding dimensions; and DoRA, which decomposes pretrained weights into magnitude and direction components, applying LoRA solely to the direction component to reduce the number of trainable parameters. While these methods are not primarily designed to mitigate catastrophic forgetting and may not be directly applicable to MCIT, some of their underlying principles may offer valuable insights. We have therefore included a discussion of these approaches in a separate subsection within the Related Work section for readers’ reference.

Incomplete Definition of the Learning Objective

The original loss function is the standard language modeling loss, i.e., next-token prediction loss, consistent with the base model (LLaVA-1.5). It is defined as:
$L_{lm} = - \sum_{i=1}^{L} \log P(x_i | X_v, X_\text{instruct,<i}, X_{a,<i}),$ where $L$ is the length of the answer, $X_v$ denotes the visual input tokens, $X_\text{instruct,<i}$ and $X_{a,<i}$ represent the instruction and answer tokens prior to the current prediction token $x_i$, respectively.
During training on the first task, the objective is solely this language modeling loss. For subsequent tasks, the total loss becomes a combination of the language modeling loss and the regularization loss $L_{reg}$ as defined in Eq. 2 of the paper: $L_{total} = L_{lm} + L_{reg}.$
We will incorporate this complete formulation and explanation into the revised version.

Adding a Related Works Section in the Main Text

Thank you for the helpful suggestion. As recommended, we will add a concise Related Works section to the main text, include the introduction to LoRA and a brief review of MCIT works within it, and retain the more detailed review of related literature in the appendix.

Bounding Box Responses under Superficial Forgetting

Your idea is compelling, but it is challenging to identify samples in our existing dataset that match the scenario you described—where both the question and the answer contain an object present in the image. To investigate this, we manually constructed a few examples. Our observations indicate that the model’s predicted bounding boxes tend to align with the object mentioned in the question, rather than the one in the answer. This behavior reflects an inductive bias introduced by the grounding task, in which training samples typically require the model to localize the object referenced in the question. These findings suggest that superficial forgetting may lead the model to over-rely on task-specific biases, hindering its ability to adapt to the actual demands of the current query. Rather than interpreting the intent of the question, the model appears to default to habitual responses based on prior training. If the model had even partially understood the question, we would expect it to at least identify the object mentioned in the answer. This further illustrates how superficial forgetting can obscure the model’s ability to adjust its behavior based on contextual demands. Thank you for your insightful question.

Thank you for reviewing our paper and for your positive feedback on the performance and experimental validation of our proposed method. Below, we address your concerns in detail:

Unnecessary Use of InternVL2 in Certain Distractor Generation Scenarios

As you correctly noted, using InternVL2 for distractor generation is unnecessary in some cases. In the CoIN dataset, this applies to the ImageNet and Grounding tasks. Accordingly, we did not employ InternVL2 for these tasks. For ImageNet, distractors are randomly sampled from the class list excluding the ground-truth one. For Grounding, we generate random bounding boxes with Intersection-over-Union (IoU) less than 0.5 with respect to the ground-truth box.
While this special handling is briefly mentioned in Appendix C.2, a formal definition was not provided. Following your suggestion, we will incorporate a more rigorous definition based on the characteristics of the answer domain, enabling this issue to be addressed not only in CoIN but also in other benchmarks and real-world applications.

Capability of InternVL2 to Understand and Solve the Task

The ability of InternVL2 to generate explanations does not imply a full understanding of the task. This is because the model is provided with the ground-truth answer during explanation generation, eliminating the need for answer reasoning. Consequently, the model’s task is only generating an explanation conditioned on the correct answer, which is significantly easier.
To validate this point, we conducted experiments evaluating InternVL2-26B on the eight tasks in CoIN. InternVL2-26B achieves an average accuracy of 53.76%, whereas our SEFE model attains 58.57% after continual learning across all tasks. Notably, SEFE is based on a 7B model, considerably smaller than InternVL2-26B. These results demonstrate that continual training of an MLLM is meaningful because it can yield superior performance with fewer inference resources.

Modification of Fig. 3

Thank you for the suggestion. We will revise Fig. 3 to include a plus sign, clearly indicating the combination method of $R_1$ and $R_2$.

We appreciate your time and effort in reviewing our paper, as well as your recognition of our contributions. Below, we respond to your comments in detail:

Comparison with the [SP’2024] Paper

Differences between the Two Papers

Thank you for pointing out this related work. This study indeed shares our goal of understanding catastrophic forgetting, and its concept of spurious forgetting is in some respects similar to our definition of superficial forgetting. However, there are several key differences between the two studies.

• Superficial forgetting refers to cases where the model fails to generate responses in the expected format, indicating a loss of response style. In such situations, it remains unclear whether the underlying knowledge has actually been forgotten. In contrast, spurious forgetting describes scenarios in which the model loses task alignment without any genuine loss of knowledge. Since spurious forgetting can be easily recovered, it is referred to as "spurious", whereas superficial forgetting does not emphasize recoverability. Thus, these two concepts are distinct.

• The [SP’2024] paper focuses primarily on the analysis of forgetting, examining changes in gradients and feature principal components during forgetting. Although it proposes a mitigation strategy—Freeze, which freezes a few bottom layers—this approach is simple, is not the core contribution of the paper, and shows limited effectiveness (e.g., ~60% forgetting rate). In contrast, our work focuses on practical solutions. We propose Answer Style Diversification (ASD) and RegLoRA to address both superficial forgetting and essential forgetting via data reconstruction and weight update constraints. On the CoIN benchmark, our method reduces the average forgetting rate to 10.45%, substantially lower than the 60% reported in the [SP’2024] paper. While differences in datasets limit direct comparability, the trend highlights the greater effectiveness and practicality of our approach. By contrast, Freeze serves more as an analytical tool of superficial forgetting than a standalone solution.

• In addition to superficial forgetting, our study also examines essential forgetting, where the underlying knowledge is genuinely lost. In contrast, the [SP’2024] paper focuses exclusively on spurious forgetting that involves no knowledge degradation. This highlights another fundamental difference between the two studies.

• Finally, the [SP’2024] paper focuses on LLMs, whereas our work addresses forgetting in MLLMs, marking a clear difference in research scope.

Performance Comparison

We also evaluated Freeze on our benchmark, with the following results:

Here, FFT refers to full-parameter fine-tuning and serves as the baseline. While Freeze outperforms the baseline, it consistently underperforms compared to our method across all metrics. This is likely because Freeze is not well-suited for Multimodal Continual Instruction Tuning (MCIT), as it was not originally designed for this setting. Additionally, as discussed earlier, Freeze functions more as a supplementary analytical tool, so its performance may not be very satisfactory as a dedicated continual learning method.

Comparison with Other Papers

In addition to the [SP’2024] paper, we identified several other studies that investigate catastrophic forgetting. For example, [1] shows that catastrophic forgetting during LLM fine-tuning becomes more pronounced as the loss landscape sharpens, suggesting a strong positive correlation between sharpness and forgetting. [2] argues that in MLLMs, catastrophic forgetting arises as fine-tuning shifts the model’s focus from general visual-text alignment to dataset-specific overfitting, resulting in performance degradation even when the vision encoder is frozen. In contrast to these approaches, our work proposes decomposing catastrophic forgetting in MCIT into two components—superficial forgetting and essential forgetting—and addresses them separately, offering a new perspective on the problem.

As suggested, we will include a comparison between our work and other related works (such as [SP’2024], [1], [2]) for understanding catastrophic forgetting in the revised version.

[1] [EMNLP 2024] Revisiting Catastrophic Forgetting in Large Language Model Tuning
[2] [CPAL 2024] Investigating the Catastrophic Forgetting in Multimodal Large Language Models