Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models

We thank Reviewer TLHk for recognizing the importance of addressing gradient misalignment in UMGMs, and for acknowledging our novel continual learning perspective compared to concurrent architectural approaches. We are also glad the reviewer found our method effective in mitigating catastrophic forgetting in image generation during instruction tuning. The questions and suggestions raised are appreciated, and we provide detailed responses below.

Weakness 1: Forgetting caused by fine-tuning on image generation and alternating tasks

Thank you for this insightful suggestion to evaluate MoDE in two additional settings: preventing forgetting of text generation while improving image generation, and training the model in an alternating task schedule. We agree that testing for forgetting of text generation while improving image generation, as well as alternating training, are both highly relevant extensions. While we have not yet explored the full alternating schedule, we conducted a new experiment where the model is fine-tuned on image generation only, and we evaluate how much it forgets multimodal understanding. Results are shown in Table R1 below.

Table R1: Forgetting caused by fine-tuning on image generation tasks.

These results confirm that fine-tuning on image generation can cause catastrophic forgetting in multimodal understanding tasks. MoDE is able to mitigate this forgetting effectively, recovering near-zero-shot performance. This demonstrates that our method is bidirectionally effective. As for the alternating training schedule, we agree this is an important and realistic scenario. While it is out of scope for this version due to resource limitations, we are actively working on extending MoDE to this setting and will include it in future work.

Weakness 2: Proposition 1 triviality

We acknowledge that Proposition 1, which establishes the connection between gradient conflict and loss increase, may appear intuitive. However, prior works such as OmniMamba [3] and MOSS [4] have focused on architectural or adapter-level decoupling without providing a theoretical explanation for inter-modal interference. To the best of our knowledge, our analysis is the first to formally link gradient conflict to inter-modal catastrophic forgetting, offering a principled understanding of why modality interference occurs. This theoretical insight directly motivates the design of modality-decoupled adapters in MoDE. Furthermore, Appendices A and B empirically validate our motivation, showing that the theoretical predictions align closely with observed performance.

Weakness 3: Incremental core idea

We agree with the reviewer’s point that LoRA and KD have been studied individually. However, our work makes a non‑trivial advance by i) jointly integrating these components within a continual instruction‑tuning paradigm, and ii) adding a gradient‑orthogonality regularizer. Critically, the following ablations demonstrate that neither piece alone, nor a naive combination of them can yield the strong trade‑offs we report below:

1. KD without decoupling. As shown in Table R2, adding our cross‑modal KD loss to CL‑MoE or to MoELoRA reduces multimodal understanding accuracy (e.g. 34.14% to 32.41% for MoELoRA), even though it does mitigate visual‑generation forgetting. This is because without explicit modality decoupling, KD “locks in” the teacher’s features in a way that hinders the model’s ability to adapt to new multimodal understanding tasks.

Table R2: Effect of KD on MoELoRA and CL-MoE.

2. Decoupled system without KD. As shown in Table R3, decoupling textual and visual LoRA without knowledge distillation yields only modest gains in understanding accuracy and visual generation quality compared to vanilla LoRA. This indicates that decoupling alone is insufficient to maintain the rich cross-modal alignment necessary for high-quality generation. Furthermore, Section 5.3 of the main paper includes an ablation on “MoDE w/o KD” (i.e., decoupled T-MoE and V-adapter without knowledge distillation), which further confirms that knowledge distillation plays a critical role in the performance improvements observed.

Table R3: Ablation on Fully De-coupled Architecture.

The above ablations show that MoDE is more than the sum of its parts: only the combination of modality‑decoupling plus our specialized distillation leads to consistently superior trade‑offs.

Weakness 4: Concurrent works

Thank you for the valuable suggestion to discuss concurrent works on better unified modeling and to clarify whether our contributions are orthogonal to these approaches. Both BAGEL [2] and OmniMamba [3] adopt decoupled encoders, and OmniMamba further applies decoupled LoRA adapters, which have similarities to our approach. This indicates that the design of modality-decoupled experts has been recognized as an effective solution. However, their focus is primarily on architectural decoupling for pretraining or static tasks, rather than continual instruction tuning. In contrast, our work addresses the challenges of continual adaptation, which is essential for real-world scenarios such as lifelong multimodal assistants and adaptive AI systems that learn and evolve over time.

Q1: KD on top of seq LoRA

Thank you for the question regarding the effect of applying knowledge distillation (KD) on top of sequential LoRA. We conducted an ablation to evaluate the effect of adding KD on top of vanilla sequential LoRA fine-tuning. As shown in Table R4 below, applying KD leads to modest improvements in both image generation quality (higher alignment scores, lower FID) and multimodal understanding performance (higher accuracy, reduced forgetting). This suggests that KD alone can partially mitigate forgetting and preserve visual generation quality. However, as shown in other results, KD without proper modality decoupling may still interfere with understanding tasks. MoDE outperforms SeqLoRA+KD by explicitly addressing this interference through modality separation.

Table R4: Effect of knowledge distillation on SeqLoRA.

References

[1]: Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in neural information processing systems 35 (2022): 25278-25294.

[2]: Deng, Chaorui, et al. "Emerging properties in unified multimodal pretraining." arXiv preprint arXiv:2505.14683 (2025).

[3]: Zou, Jialv, et al. "Omnimamba: Efficient and unified multimodal understanding and generation via state space models." arXiv preprint arXiv:2503.08686 (2025).

[4]: Xu, Zhiyang, et al. "Modality-Specialized Synergizers for Interleaved Vision-Language Generalists." The Thirteenth International Conference on Learning Representations. 2025.

[5]: Chen, Cheng, et al. "Coin: A benchmark of continual instruction tuning for multimodel large language models." Advances in Neural Information Processing Systems 37 (2024): 57817-57840.

[6]: Huai, Tianyu, et al. "CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

We thank Reviewer LAT9 for the positive feedback on the clarity of our writing, the importance of addressing forgetting in both modalities of UMGMs, and the thoroughness of our experiments and appendix. We appreciate the constructive comments and address the raised concerns in detail below.

Weakness 1: Statistical validity of results in Tab. 2

We thank the reviewer for raising this important point of the statistical significance of the improvements over CL-MoE. To evaluate the robustness of our results, we conducted 3 runs with different random seeds for both CL-MoE and MoDE. The results are summarized in Table R1 below. We report the mean and standard deviation across 3 runs.

Table R1: Results with multiple seeds (with mean and standard deviation). For each method, the upper row shows the best accuracy during continual instruction tuning, and the lower row shows the final accuracy after completing all tasks.

Although the absolute improvement in ACC may appear small (e.g., +0.55), MoDE consistently improves both accuracy and forgetting across all seeds and datasets. Given the small standard deviations, the performance gains are meaningful and robust. These results strengthen the statistical significance and reliability of our conclusions.

Weakness 2: Introduction of V-Adapter and T-MoE

Thank you for pointing out that V-Adapter and T-MoE are not formally introduced in the main text and are only briefly mentioned in the caption of Figure 3. We agree that the current version does not clearly introduce V-Adapter and T-MoE in the main text. In the revised version, we will explicitly define them in Section 4.2 Modality-Decoupled Experts (MoDE).

Weakness 3: Ablation on the number of experts in T-MoE

Thank you for the suggestion. We provide an ablation study on the number of experts in the T-MoE module. As shown in Table R2 below, increasing the number of experts from 4 to 6 yields slightly improved text alignment and accuracy, though the gains are modest. This suggests that the performance of MoDE is relatively robust to the number of experts, and 4 experts offer a good trade-off between performance and computational cost.

Table R2: Ablation on the number of experts in T-MoE.

Q1: Time and Memory Costs Compared to CL-MoE

Thank you for the question of "How does MoDE compare to CL-MoE regarding time and memory costs?". We compare MoDE with CL-MoE and MoELoRA in terms of TFLOPs, peak GPU memory, training time per iteration, and the percentage of trainable parameters. Results are shown in Table R3 below. MoDE incurs a slightly higher computational cost and memory usage than CL-MoE due to its use of both modality-specific adapters and knowledge distillation. However, the differences are modest and well within practical bounds. In return, MoDE consistently yields better retention and overall performance, demonstrating a favorable trade-off between efficiency and effectiveness.

Table R3: Comparison of training efficiency and memory usage.

Q2: Reference dataset

Thank you for the question regarding the composition of the reference dataset used for knowledge distillation (KD). The reference dataset used for KD loss is composed of text prompts sampled from the LAION-5B dataset [1]. The dataset is shared across all tasks, rather than being tailored to the current task. The shared reference dataset ensures that the KD signal reflects general visual generation capability, independent of the downstream multimodal understanding task. We will clarify this more explicitly in the revised manuscript.

References

[1]: Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in neural information processing systems 35 (2022): 25278-25294.

[2]: Huai, Tianyu, et al. "CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[3]: Chen, Cheng, et al. "Coin: A benchmark of continual instruction tuning for multimodel large language models." Advances in Neural Information Processing Systems 37 (2024): 57817-57840.

We thank Reviewer qoBP for highlighting the novelty of our work as the first to study catastrophic forgetting of image generation in UMGMs under continual instruction tuning, and for appreciating our theoretical explanation based on gradient conflicts between image and text generation losses. The questions raised are insightful, and we provide our detailed responses below.

Weakness 1: Fine-tuning on image generation tasks

Thank you for the insightful question: whether fine-tuning on image generation tasks would lead to catastrophic forgetting of the understanding capabilities, and whether our proposed method can mitigate such forgetting. To address this, we conduct new experiments by fine-tuning the linear layers of the Chameleon model [3] on image generation tasks. Specifically, we use the LAION-5B dataset [1] for fine-tuning and apply knowledge distillation using the multimodal understanding sequence from the GQA dataset [2] as reference data. We compare our MoDE method with LoRA fine-tuning across three multimodal understanding benchmarks. Accuracies are reported in Table R1. The difference between zero-shot and fine-tuned performance reflects the degree of forgetting in multimodal understanding.

Table R1: Forgetting caused by fine-tuning on image generation tasks.

These results support two conclusions:

(1) Yes, fine-tuning on image generation tasks can indeed cause catastrophic forgetting in multimodal understanding, confirming that the forgetting problem is bidirectional. This is evident in the substantial drop in performance for the LoRA baseline compared to zero-shot.

(2) Our MoDE method remains effective in this scenario, significantly reducing the performance drop and mitigating forgetting relative to the LoRA baseline. This demonstrates that MoDE generalizes beyond the specific forgetting direction explored in the main paper, where understanding tasks overwrite prior image generation capabilities, and is effective regardless of the fine-tuning direction.

Weakness 2: "Reference Data" in Figure 3

In our experiments, "Reference Data" are text prompts selected from LAION-5B dataset [1].

Q1: CoIN benchmark

Thank you for pointing this out. Due to resource constraints, we selected a subset of 6 tasks from the full CoIN benchmark to ensure feasibility while maintaining diversity across modalities and task types. Specifically, the chosen tasks (ScienceQA, TextVQA, ImageNet, GQA, VizWiz, and Grounding) span a representative range of vision-language reasoning, recognition, and grounding challenges. While we did not include VQAv2 and OCR-VQA in our current evaluation, we expect similar trends to hold, given the consistency of results across the six evaluated tasks. Extending evaluation to all 8 tasks is a valuable contribution and we plan to include it in the final paper.

Q2: Additional supplementary results

Thank you for the helpful suggestion regarding the missing results for DualPrompt and MoELoRA under different task orderings, as well as the missing Janus-Pro results for several baselines in the supplementary material. We provide the complete results below:

Q2.1: We report additional ablations on task order sensitivity in Table R2. DualPrompt [5] and MoELoRA [4] are now included alongside MoDE. DualPrompt yields low accuracy and low forgetting. This is because adding a small set of prompt embeddings slightly biases the feature representations, which is insufficient to learn the new and hard tasks. The results in Table R2 show that MoDE maintains superior performance and robustness under alternate task orders.

Table R2: Additional ablation on task order.

Q2.2: In Table R3, we provide the missing Janus-Pro results for Model Tailor [6], MoELoRA[4], and our MoDE. Again, MoDE demonstrates the best performance with the lowest forgetting and highest accuracy.

Table R3: Evaluation on Janus-Pro benchmark.

Q3: KD on MoELoRA and CL-MoE

Thank you for the thoughtful question about the effect of applying image generation knowledge distillation to MoELoRA and CL-MoE, and whether MoDE’s benefit extends beyond that. To directly evaluate the effect of applying knowledge distillation (KD) for image generation on existing baselines, we conduct new experiments by incorporating KD into MoELoRA and CL-MoE. The results are shown in Table R4.

Table R4: Effect of KD on MoELoRA and CL-MoE.

These results show that applying KD improves image generation quality for both MoELoRA and CL-MoE, as indicated by higher alignment scores and lower FIDs. However, KD degrades the multimodal understanding performance (reducing accuracy and increasing forgetting) because the models are not decoupled, and the KD interferes with learning on understanding tasks. In contrast, MoDE achieves the best performance across all metrics by design: its modality-decoupled structure prevents interference from KD and enables effective retention of multimodal understanding.

Q4: Bold fonts

Thanks for valuable suggestion and we will bold the best results for every metric.

References

[1]: Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in neural information processing systems 35 (2022): 25278-25294.

[2]: Hudson, Drew A., and Christopher D. Manning. "Gqa: A new dataset for real-world visual reasoning and compositional question answering." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[3]: Team, Chameleon. "Chameleon: Mixed-modal early-fusion foundation models." arXiv preprint arXiv:2405.09818 (2024).

[4]: Chen, Cheng, et al. "Coin: A benchmark of continual instruction tuning for multimodel large language models." Advances in Neural Information Processing Systems 37 (2024): 57817-57840.

[5]: Wang, Zifeng, et al. "Dualprompt: Complementary prompting for rehearsal-free continual learning." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[6]: Zhu, Didi, et al. "Model tailor: Mitigating catastrophic forgetting in multi-modal large language models." arXiv preprint arXiv:2402.12048 (2024).

[7]: Huai, Tianyu, et al. "CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

We thank Reviewer vBLd for the positive assessment of our paper’s theoretical contributions, strong empirical results in continual instruction tuning, and the well-motivated study of intra- and inter-modal forgetting in unified multimodal models. The questions raised are insightful, and we will address each point below to clarify our contributions and the specifics of our work.

Weaknesses 1&2, Q2: Novelty and decoupled baselines

We understand the concern that our building blocks (MoE‑LoRA, LoRA, KD) have appeared in prior work. To demonstrate that their combination in MoDE is neither trivial nor interchangeable, we conducted the following ablations:

1. KD without decoupling. As shown in Table R1, adding our cross‑modal KD loss to CL‑MoE or to MoELoRA reduces multimodal understanding accuracy (e.g. 34.14% to 32.41% for MoELoRA), even though it does mitigate visual‑generation forgetting. This is because without explicit modality decoupling, KD “locks in” the teacher’s features in a way that hinders the model’s ability to adapt to new VQA and grounding tasks.

Table R1: Effect of adding KD to MoELoRA and CL-MoE.

2. Decoupled system without KD. As shown in Table R2, decoupling textual and visual LoRA without knowledge distillation yields only modest gains in understanding accuracy and visual generation quality compared to vanilla LoRA. This indicates that decoupling alone is insufficient to maintain the rich cross-modal alignment necessary for high-quality generation. Furthermore, Section 5.3 of the main paper includes an ablation on “MoDE w/o KD” (i.e., decoupled T-MoE and V-adapter without knowledge distillation), which further confirms that knowledge distillation plays a critical role in the performance improvements observed.

Table R2: Ablation on Fully De-coupled Architecture.

In conclusion, the above ablations show that MoDE is more than the sum of its parts: only the combination of modality‑decoupling plus knowledge distillation leads to consistently superior performance.

Weakness 3 & Q1: Parameter and computational costs comparisons

We appreciate the reviewer’s suggestion. To address concerns regarding the triviality of our method, we report a comparison of training compute, memory footprint, and trainable parameters in the table below. As shown, MoDE introduces only a slight increase in training time (440.3 ms vs. 419.2 ms) and peak GPU memory (84.98 GB vs. 75.27 GB) compared to MoELoRA, while maintaining comparable TFLOPs. The trainable parameter ratio of MoDE is only 0.0211%, modestly higher than MoELoRA and CL-MoE (0.0171%), indicating that our performance gains are not simply due to increased processing capacity. These results suggest that MoDE's improvements arise from its design (modality-decoupled experts and distillation), rather than significantly higher resource usage.

Table R3: Comparison of Trainable Parameters and Training Costs.

Weakness 4: Text-only inter-modal forgetting

We appreciate the reviewer’s insightful comment and references. This is the first work on inter-modal forgetting on UMGMs continual learning, so we use the forgetting in visual generation caused by continual instruction tuning on multimodal understanding as the study case. This is a very interesting direction; we will explore it in our future work. However, this is currently beyond the scope of our study, which aims to establish the foundational analysis and mitigation strategies for inter-modal interference through the lens of visual generation degradation.

Weakness 5: Correction of terminology

We apologize for the confusion caused by the misuse of "text-only generation" and "language generation" and will replace them with "multimodal understanding" in the revised paper.

Q3: Figure 1 explanation

The question raised concerns about whether Figure 1 uses LoRA adapters to measure forgetting, and why forgetting caused by full-model fine-tuning was not evaluated. Yes, Figure 1 is obtained using LoRA-based fine-tuning. While we acknowledge that full-model fine-tuning may lead to different absolute values, we chose LoRA due to practical constraints on compute resources and because it has become a widely used paradigm for efficient continual adaptation in large-scale models. Importantly, our goal is to study the modality imbalance and forgetting phenomena under realistic fine-tuning setups. LoRA provides a representative and resource-efficient way to examine these effects in current practice. Moreover, prior work (e.g., [1]) has shown that full fine-tuning tends to exacerbate catastrophic forgetting in MLLMs, so our reported forgetting levels are likely conservative. Therefore, even under LoRA, the modality-specific forgetting patterns are clearly observable and support the motivation and effectiveness of our proposed MoDE approach.

References

[1]: Zhu, Didi, et al. "Model tailor: Mitigating catastrophic forgetting in multi-modal large language models." arXiv preprint arXiv:2402.12048 (2024).

[2]: Chen, Cheng, et al. "Coin: A benchmark of continual instruction tuning for multimodel large language models." Advances in Neural Information Processing Systems 37 (2024): 57817-57840.

[3]: Huai, Tianyu, et al. "CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.