Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning

We sincerely thank the reviewer for the helpful and encouraging comments. We appreciate that they note the inclusion of theoretical analysis, find our writing clear and easy to read, and recognize the significance of the problem setting. We hope the following explanations provide sufficient clarification.

---

Q1. Tasks-Architecture Conflict in MLLMs

Thank you for the question. Our intention here is to emphasize why fixed architectures pose additional challenges for continual learning (CL) in MLLMs. We would like to clarify it further here.

MLLMs have inherent architectural heterogeneity, especially in the commonly adopted vision encoder + projector + LLM framework. Unlike unimodal LLMs, MLLMs process inputs from multiple modalities. As tasks vary in their reliance on each modality, abstraction requirements differ across modules. This leads to greater task-wise architectural sensitivity during CL. As shown in Figure 1, the sensitivity of transformer layers in both the vision encoder and the LLM varies substantially across tasks, indicating that different components must adapt differently. This observation motivates us to move beyond fixed architectures and adopt dynamic expert allocation, and further supports our design of an architecture evolution mechanism tailored to the evolving demands of CL in MLLMs.

We will revise the Introduction to better explain this motivation.

---

Q2. Threshold Selection and Sensitivity

Thank you for the question. The threshold in Equation 9 is selected based on the reconstruction loss distribution of each autoencoder on its corresponding task’s training data. It is set moderately above the typical loss range to accommodate minor distribution shifts at evaluation and to allow transferable samples from other tasks to activate relevant experts.

We observe that the loss distributions are concentrated with few outliers, and the same thresholding strategy works across tasks without tuning. Moreover, in-task and out-of-task losses are typically well-separated, as supported by the t-SNE visualization in Figure 5.

The threshold serves two purposes: (1) avoiding irrelevant experts from being activated solely due to top-2 ranking, and (2) enabling detection of unseen tasks. As such, it plays a filtering role rather than directly driving routing decisions. This allows for coarse-grained threshold choices without requiring precise tuning.

To assess sensitivity, we evaluate the final checkpoint of D-MoLE under different scaling factors of the thresholds $\\{\tau_t\\}$ and report the corresponding average Last scores across tasks, with $1\times$ denoting the default setting. Full table can be found at https://anonymous.4open.science/r/D-MoLE/table10.jpg.

Table 1: Average task performance under different threshold scaling factors, evaluated on the final checkpoint.

Note that this experiment applies threshold scaling only during evaluation. Due to current computational resource constraints, we do not retrain the model under each new threshold setting. As a result, expert collaboration patterns during evaluation may differ from those formed during training under the default thresholds, which may slightly affect performance. Despite this, performance remains stable across a wide range of threshold scaling factors, indicating that our method is not sensitive to the exact threshold values, as long as they are within the same order of magnitude. We will include this analysis in the revised version.

---

Q3. Setup of Figure 5 and Autoencoder Capacity

Thank you for pointing this out. The experimental setup for Figure 5 (Appendix G) is as follows: we randomly sample 500 training samples from each task and extract their multimodal instruction sequence embeddings using the method described in Section 4.1. We then compute the reconstructed embeddings using the corresponding autoencoders and visualize all embeddings together using t-SNE. The clear separation among clusters provides supporting evidence that task-specific autoencoders learn distinct representations and can effectively function as routers. We will include these experimental details in the revised version for clarity.

Regarding the use of a 2-layer MLP (one encoder and one decoder layer) for the autoencoder, we find it sufficient in our setting. In typical CL benchmarks for MLLMs, tasks are defined by dataset boundaries, and samples from different tasks often differ significantly in terms of image domains, question types, or textual styles. As such, the autoencoders can effectively distinguish tasks even with simple architectures. We adopt this lightweight design to minimize the additional computational overhead introduced by the routing process. Exploring more challenging scenarios with blurred task boundaries may be a promising direction for future work.

We sincerely thank the reviewer for the encouraging and constructive feedback. We are pleased that they find our approach well-motivated, intuitive, and effective, and acknowledge the impressive performance gains. We hope our responses below address the remaining concerns.

---

Q1. Including Some Case Studies or Examples

Thank you for the valuable suggestion. In the revised version, we will include case studies in the appendix. For example, we plan to show some multimodal instruction examples from early tasks and compare the model’s responses to them after training on later tasks. This will help illustrate how well D-MoLE preserves prior knowledge by examining model behavior on early-task examples at different stages of continual learning.

---

Q2. Figure Clarity and Font Size

Thank you for the helpful suggestion. We will increase the font size in Figure 1 to improve readability, and recheck all figures in the paper to ensure visual clarity in the revised version.

---

Q3. Visualizing the Dynamic Training Process

Thank you for the valuable suggestion. We have created two heatmaps to illustrate the dynamic behavior of our method and better convey the adaptability of D-MoLE. We provide preview versions at anonymous links below, and will incorporate them into the revised version:

• Architecture evolution dynamics (https://anonymous.4open.science/r/D-MoLE/figure6.jpg): shows how experts are allocated across different layers during the CMIT process.
• Expert activation dynamics (https://anonymous.4open.science/r/D-MoLE/figure7.jpg): visualizes how task-specific experts are activated over time during training.

---

Q4. Compatibility with Other Continual Learning Approaches

Thank you for the question. Combining our method with other continual learning approaches is certainly feasible.

Our method focuses on architectural evolution, dynamically expanding model capacity throughout the CMIT process. This enables flexible task adaptation without being constrained by a fixed parameter budget. In contrast, approaches like O-LoRA mitigate forgetting through parameter regularization, but often face a trade-off between retaining prior knowledge and adapting to new tasks due to their fixed capacity.

These two directions are compatible in principle. While our current framework does not include regularization terms in the loss function, incorporating them may be a promising extension. For example, enforcing orthogonality between different experts could help reduce redundancy and improve task separation.

---

Q5. Need for Expert Removal

Thank you for the question. While our current experimental setting does not involve expert removal, since we evaluate all tasks after each training stage to assess knowledge retention (see Appendix F), our framework can easily support it.

For example, one practical extension is to track expert activation frequency using a sliding window over recent inputs. Experts associated with inactive tasks could then be unloaded to reduce memory and computation overhead, with the option to reload or reinitialize them if needed. This may be a promising direction for future work, particularly in real-world online continual learning scenarios.

We sincerely thank the reviewer for the thoughtful and positive feedback. It is encouraging that they find our paper well-written and easy to follow, our results impressive, and our method practical, intuitive, and concise. We hope the clarifications below address the reviewer’s remaining concerns.

---

Q1. Clarification of the Two 'Average' Metrics in Table 2

Thank you for the valuable suggestion. To clarify:

• The Average column (rightmost) shows the mean performance of each method across all datasets (i.e., row-wise average).
• The Average row near the top refers to one of the continual learning (CL) metrics introduced in Section 5.1, alongside Last and BWT. Their formal definitions are provided in Appendix F.

To improve clarity, we will consider renaming the CL-specific Average row to AVG in the revised version.

---

Q2. Unexpectedly Strong Seq-FT Performance on KVQA

Thank you for pointing out this observation. Seq-FT achieves relatively high accuracy on KVQA immediately after training and experiences minimal forgetting after training on PMC-VQA. These two factors together explain its higher Last and BWT scores on KVQA.

We attribute this behavior to two possible factors. First, KVQA is a relatively difficult dataset, where most methods show only modest improvements over the zero-shot baseline. CL methods that incorporate regularization mechanisms (e.g., LwF-LoRA, EWC-LoRA) may struggle to fully adapt to such tasks due to limited flexibility, whereas Seq-FT’s unconstrained fine-tuning can more easily fit the data. Second, PMC-VQA is a medical-domain task with different visual domains, question types, and prompt formats compared to KVQA. This reduces representational overlap between the two tasks and thus limits interference during sequential training. However, regularization-based methods may apply global constraints which could result in unintended interference with prior task-specific adaptation. A similar pattern is observed between VizWiz-Cap and SK-VG, where Seq-FT also exhibits reduced forgetting, likely due to the same reason.

Overall, while Seq-FT achieves higher Last and BWT scores on KVQA, the margins are small. Across the full task sequence, it still suffers from severe forgetting, as reflected in its low overall CL scores.

---

Q3. Strong Performance on the Initial Task

Thank you for the thoughtful question. Our training protocol treats all tasks in the sequence uniformly and does not apply any special treatment to the initial task, so there is no risk that our method simply overfits to the first dataset. As evidenced by the overall experimental results, our method outperforms all state-of-the-art baselines on all three CL metrics across most tasks, not just the initial one.

The seemingly large performance gains on VizWiz-Cap can be explained as follows:

• Following the CoIN benchmark [1], we use CIDEr as the evaluation metric for captioning tasks. Since CIDEr scores are not upper-bounded (often exceeding 1), numerical improvements may appear larger in magnitude.
• In continual learning, forgetting accumulates over time and amplifies performance degradation on earlier tasks. As D-MoLE better mitigates forgetting, its advantage is more visible on tasks like VizWiz-Cap that appear early in the sequence.

---

Q4. Key Factors Driving Performance Improvements

Thank you for the question. The performance gains of D-MoLE stem from the integration of two complementary modules, each addressing a key challenge in CMIT. The dynamic layer-wise expert allocator addresses task architecture conflict by assigning LoRA experts only to the most relevant layers and routing inputs via task-specific autoencoders. This enables precise architectural adaptation and selective expert activation during inference. The gradient-based inter-modal continual curriculum mitigates modality imbalance by adjusting training dynamics when tasks rely unevenly on different modalities, ensuring stable multimodal adaptation.

Our ablation study (Section 5.3) supports this analysis. Removing the expert allocator (v4) leads to the largest performance drop, underscoring the importance of architectural flexibility. Removing the curriculum module (v3) results in lower performance than the full model, demonstrating its complementary benefit. Limiting updates to a single modality (v1 or v2) also yields suboptimal results, highlighting the need for full-modality adaptation.

Overall, while the expert allocator contributes most directly to performance gains, both components are essential and jointly enable robust and balanced continual learning. We will add more detailed discussion of these factors in the revised version.

---

Reference

[1] CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models. NeurIPS 2024.

We sincerely thank the reviewer for the thoughtful and encouraging feedback. We are glad that they find our paper well-structured and our approach to be inspiring. We hope our responses below help clarify the remaining points.

---

Q1. Notation Refinement

Thank you for the valuable suggestion. We will revise the notation to use bold symbols for vectors (e.g., $\mathbf{x}$) and standard math notation for scalars (e.g., $x$) to improve clarity.

---

Q2. Difference Between Finetune and Seq-FT in Table 2

Thank you for the thoughtful question. The Finetune results refer to models trained independently on each task from scratch using LoRA fine-tuning, and evaluated only on that task. These results serve as upper bounds without any continual learning (CL) constraints. In contrast, Seq-FT is a vanilla CL baseline that performs sequential LoRA fine-tuning across the task sequence, without any specific mechanism to mitigate forgetting. As expected, it performs poorly on all three CL metrics.

All CL methods (i.e., all rows except Zero-shot, Finetune, and Joint-learning) are evaluated under the same protocol: after training each new task, the model is tested on all tasks in our benchmark. This differs from traditional CL setups, which typically evaluate only on seen tasks. Since the pretrained MLLM has zero-shot capabilities, our protocol also assesses how well this ability is retained. The details of this evaluation protocol can be found in Appendix F. We will clarify this distinction in the revised version.

---

Q3. Training Efficiency Despite Architecture Evolution

Thank you for noticing this point. We also observe that D-MoLE brings an additional benefit in training efficiency, as shown in Table 6. While we briefly discussed this below the table, we would like to elaborate on it here.

The main source of this efficiency gain lies in the selective placement of LoRA modules. Unlike methods such as joint-learning and O-LoRA, our method inserts LoRA modules only into a subset of the most sensitive transformer layers for each task. This design ensures that, although our method slightly increases the LoRA rank, the number of trainable parameters remains comparable to these baselines due to fewer insertion points.

To further illustrate the efficiency, we conduct a toy experiment comparing GPU runtime under different LoRA ranks. Specifically, we generate a random input matrix of shape 64 × 1024 and perform 10,000 iterations of multiplying it with two simulated LoRA modules of rank 4 and rank 8, respectively. The measured average runtime per iteration is summarized below:

Table 1: GPU runtime comparison for different LoRA ranks.

Despite doubling the rank, the runtime increases by only 6.8%, suggesting that in low-rank settings, the startup overhead of matrix multiplication dominates the total runtime rather than the rank itself. Since our method inserts LoRA modules into fewer transformer layers, the total number of such operations is reduced. This compensates for the slightly higher cost per operation and results in a modest overall speedup.

Moreover, our architecture evolution mechanism is lightweight by design. As shown in Table 7, the preprocessing time for computing the zero-cost proxy accounts for less than 2% of the total training time.

We will make this explanation clearer in the revised version.

---

Q4. Load Balancing in Expert Routing

Thank you for the question. We do not explicitly introduce load balancing mechanisms in our routing design. Our routing mechanism is primarily intended for knowledge retention and transfer in the CL setting, rather than for inference efficiency or capacity scaling as in traditional MoE models.

In our framework, expert collapse is unlikely to occur. During training, task IDs are known and each corresponding expert is explicitly activated and updated. During evaluation, task-level routing is based on reconstruction losses from task-specific autoencoders, each trained on its own task data. As shown in Appendix G, the reconstructed embeddings form well-separated clusters, enabling reliable expert selection and preventing collapse. The router operates on the entire multimodal instruction sequence, rather than on individual modalities. Therefore, varying modality distributions do not directly affect expert activation.

In real-world deployments, especially under online CL settings, load balancing may become important. For example, some tasks may dominate the input stream, leading to expert overuse, while others remain underutilized. One possible solution is monitor expert activation frequency and incorporate regularization or usage-aware training objectives to maintain stable generalization. These strategies are orthogonal to our current design and may be worthwhile to explore in future work.