Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning

We sincerely thank the reviewer for their thoughtful comments and careful reading. Below we respond to each concern in detail.

---

W1: Whether the theoretical approximations—such as Taylor expansion, Fisher approximation, and symmetric finite differences—are valid, and whether minimizing the upper bound brings empirical benefits.

1. We would like to clarify that the assumptions we adopt—such as approximating the Hessian with the empirical Fisher Information Matrix, applying a second-order Taylor expansion of the loss, and neglecting higher-order terms—are not unique to our work. These are widely used and well-accepted approximations in the continual learning [1][2][3].

Such approximations have been extensively validated and empirically supported in recent deep learning theory and algorithms. Our use of them follows established practices rather than introducing new or unverified assumptions.

2. To empirically evaluate the benefit of minimizing the upper bound (Eq. 11), we conducted ablation experiments:

• LoRA-M (with merging but no perturbation) vs LoRA-P&M (with perturbation): The latter consistently improves performance, as shown in Table 4 and Figure 1.
• Additionally, in Figure 2, we visualize the loss surface across merged models. The optimal α derived from the upper-bound formulation consistently lands in low-loss regions.
• We also show in table below that adding our perturbation to external merging methods (e.g., CoFiMA(CoFiMA-P)) improves performance—indicating that minimizing the derived upper bound brings empirical benefits beyond our method.

While the approximations are imperfect, the combined analytical formulation and empirical outcomes strongly suggest that they are effective and valid in practice.

---

W2: The related work section is brief. Some early studies on continual learning and model merging are neither discussed nor compared experimentally.

Thank you for pointing this out. We agree that the related work section can be expanded. In our revised version, we will add discussion of classical continual learning methods such as EWC, GEM, and cite early model merging works then discuss distinctions from our coefficient-guided merging.

Furthermore, we have included experimental comparisons with EWC in our updated results. The results show that LoRA-P&M substantially outperforms EWC.

We also conducted additional comparisons with several recent continual learning methods:

---

W3: The computation and storage of the Fisher matrix require significant resources. Is the approach inherently limited to LoRA-based fine-tuning?

We appreciate the reviewer’s concern and would like to clarify that while our method is currently implemented with LoRA-based fine-tuning, it is not inherently restricted to LoRA. The core of our approach—the task vector merging strategy and curvature-guided perturbation—can be applied to more PEFT methods, including but not limited to adapters, prompt tuning.

In our setup, the actual overhead is modest for the following reasons:

We only compute and store the diagonal Fisher matrix for modules equipped with LoRA, which, in our implementation, are limited to the key and value projections of each attention module. These parameters account for approximately one-sixth of the total model parameters. Moreover, in practice, since the diagonal Fisher matrix is only used at the end of training for each task, it can be stored on the CPU or disk, preventing linear growth in GPU memory usage.

As shown in Table 7, the overall memory increase from LoRA(22.80GB) to LoRA-P&M(24.29GB) is approximately 1.5 GB for a dataset with 10 tasks.

While Section 3.4 introduces LoRA primarily as a means to reduce memory overhead, we emphasize that LoRA is also better aligned with the goals of continual learning (CL) beyond efficiency. LoRA tends to cause less interference with previously learned tasks[1], allows for easier task-specific adaptation, and requires fewer training examples and parameters to achieve strong performance. In contrast, full fine-tuning often leads to greater forgetting and lacks modularity. Therefore, even when memory is not a limiting factor, LoRA remains a more suitable and effective paradigm for continual learning.

---

Q1&Q3: Typos

Thank you for this sharp observation. We will correct this in the revision.

---

Q2: In Figure 2, the average loss landscape is visualized. It is unclear how this figure was constructed—what exactly does the "average" on the loss axis represent?

Thank you for pointing this out. In Figure 2:

• The “average loss” refers to the mean task-wise cross-entropy loss computed on validation data from all tasks seen up to that point.
• Each point on the 2D plane corresponds to a convex combination of $\hat{\theta}_{t-1}$ and $\theta^*_t$, i.e., $\theta = \beta \hat{\theta}_{t-1} + \alpha \theta^*_t$.
• We evaluate the merged model at that point and compute its average loss across all past tasks.

We will clarify this in the figure caption and the main text.

---

We appreciate the reviewer’s close reading and constructive feedback. The points raised will be addressed thoroughly in the revised version, and we thank you again for helping us improve the clarity and rigor of the work.

[1] James Kirkpatrick, et al. "Overcoming Catastrophic Forgetting in Neural Networks." 2017.

[2] Zhenyi Wang, et al. "A UNIFIED AND GENERAL FRAMEWORK FOR CONTINUAL LEARNING." ICLR24.

[3] Mei Li, et al. "BECAME: BayEsian Continual Learning with Adaptive Model MErging." ICML25.

[4] Xiao Wang, et al. "Orthogonal Subspace Learning for Language Model Continual Learning." EMNLP23.

[5] Jingyang Qiao, et al. "Prompt Gradient Projection for Continual Learning." ICLR24.

[6] Zhanxin Gao, et al. "Consistent Prompting for Rehearsal-Free Continual Learning." CVPR24

[7] Dan Biderman, et al. "LoRA Learns Less and Forgets Less." TMLR24.

We sincerely thank the reviewer for their insightful and detailed feedback. Below we address each point individually.

---

W1: Comparison with methods include EWC and GEM.

We have now included experimental comparisons with EWC in our updated evaluations. Since our method also leverages Fisher-based curvature approximation, we believe this is a meaningful baseline. The results show that LoRA-P&M significantly outperforms EWC across all benchmarks.

As for GEM, we respectfully did not include it because it is based on experience replay, which violates the rehearsal-free continual learning assumption adopted in our work. Our method, like SD-LoRA and InfLoRA, operates under a constraint that previous data cannot be revisited. Thus, GEM is not directly comparable in our setting.

---

W2: A detailed comparison with EWC.

We would like to clarify that our two-stage framework is conceptually distinct from EWC. Our approach introduces a model merging-based inference paradigm, where a closed-form merging coefficient $\alpha_t^*$ is derived to minimize total loss increase across all tasks.
In contrast, EWC adds a Fisher-weighted regularization term during training to penalize deviations from previously important parameters. In EWC, the Fisher matrix serves to estimate the importance of each parameter for past tasks. In our case, the Fisher matrix is used solely as a tractable approximation of the Hessian in order to compute an optimal merging direction. Thus, while we share a technical approximation with EWC, our theoretical formulation, implementation, and application context are entirely different.

We appreciate the reviewer’s observation regarding the connection to EWC. While both our method and EWC utilize the Fisher Information Matrix, the motivation and role of this approximation are fundamentally different. We will clarify this distinction in the revised version and cite EWC appropriately.

---

W3: The expression of "optimal" in the paper is not sufficiently rigorous.

Thank you for highlighting this. Our merging coefficient $\alpha_t^*$ is theoretically optimal under the Fisher-based second-order loss approximation, but we agree that this does not imply global optimality with respect to the original loss surface.

To prevent misunderstanding, we will revise our language throughout the paper (including the abstract) to refer to α as “analytically derived under a curvature approximation”, and clarify its theoretical basis and limitations.

---

W4: To what extent does the task order influence the performance of our method? How would the results change if all sub-models were initialized from the same base model rather than sequentially?

1. Indeed, task order can influence performance in continual learning. In all of our experiments, we train three runs under different random seeds, each corresponding to a different task order. We report mean and standard deviation to capture this variability. We will clarify this in the experimental setup.

2. We agree this is an important baseline. In fact, Table 3 in our paper already includes such methods:

• Model Averaging: directly merges task-specific models trained from the same base model.
• DARE: a recent data-free merging method.

---

W5: The memory grows linearly with the no of tasks, anyway to reduce this?

As you noted, storing one Fisher matrix per task results in linear memory growth. However:

• We only store diagonal Fisher approximations for low-rank LoRA modules, making the per-task cost minimal (Table 7 shows ~1.5GB/24.29GB across 10 tasks). Moreover, in practice, since the diagonal Fisher matrix is only used at the end of training for each task, it can be stored on the CPU or disk, preventing linear growth in GPU memory usage.
• Inspired by Online EWC [1], we plan to replace per-task Fisher storage with a simple running average over past tasks, resulting in a fixed-size Fisher representation. We denote this variant as Online P&M, and provide experimental comparisons against our original P&M framework.

In the 5-task and 10-task settings, the approach of aggregating Fisher information from all previous tasks performs comparably to our LoRA-P&M method. However, when the task sequence becomes longer (e.g., 20 tasks), its performance degrades significantly. Addressing this limitation is one of our directions for future research.

---

W6: The experiment setup is not clear. Is the task ID known at inference (multi-head or single-head evaluation).

We follow the class-incremental learning (CIL) setting, where the task ID is not provided at inference time, and every tasks has a classifier head. Each task expands the output space (e.g., from 20 classes to 200 classes across 10 tasks in CIFAR100), and the model must distinguish among all seen classes.

This is considered the most challenging and realistic continual learning setting, and aligns with the evaluation protocols used in SD-LoRA, CODA-Prompt, and other recent works. We will clarify this explicitly in the revised experimental setup.

---

We again thank the reviewer for their constructive suggestions. We will revise the manuscript to reflect the above clarifications, add missing citations, and expand our baseline coverage accordingly.

[1] Ferenc Huszar, et al. "On Quadratic Penalties in Elastic Weight Consolidation." 2017.

We sincerely thank the reviewer for their thoughtful and constructive feedback. Below we address each point in detail and clarify the concerns.

---

W1: LoRA-P&M requires storing a Fisher matrix for each task, leading to memory growth that is linear in the number of tasks.

In our setup, the actual overhead is modest for the following reasons:

We only compute and store the diagonal Fisher matrix for modules equipped with LoRA, which, in our implementation, are limited to the key and value projections of each attention module. These parameters account for approximately one-sixth of the total model parameters. Moreover, in practice, since the diagonal Fisher matrix is only used at the end of training for each task, it can be stored on the CPU or disk, preventing linear growth in GPU memory usage.

As shown in Table 7, the overall memory increase from LoRA(22.80GB) to LoRA-P&M(24.29GB) is approximately 1.5 GB for a dataset with 10 tasks.

---

W2: Comparison with more continual learning methods

We appreciate the reviewer pointing out recent methods. Due to time constraints, we were able to compare against O-LoRA, Prompt Gradient Projection(PGP), and Consistent Prompting(C-Prompt). These represent strong and recent methods across both LoRA-based and prompt-based paradigms.

Regarding I-LoRA and PromptFusion: these methods rely on data replay, which fundamentally differs from our task setting (rehearsal-free continual learning). As such, we respectfully argue that they are not directly comparable to our approach.

We will add the comparisons and discussion with these methods to the revised version of our paper.

---

W3: Clarification of Equation 17

Thank you for the helpful comment. We agree that the current form of Equation 17 may lead to confusion. Specifically, it may be misinterpreted as a global parameter-wise summation, whereas in practice, each LoRA update is applied locally and independently at specific network locations (e.g., attention key or value projections).

To clarify this, we will revise Equation 17 as:

$\theta_t = \theta_{t-1} + \bigoplus_{p \in l} A_t^{(p)} B_t^{(p)}$

where $\bigoplus$ denotes location-wise module insertion, not element-wise addition. Each low-rank update $A_t^{(p)} B_t^{(p)}$ is injected into the pre-trained weight at its corresponding module $p$, and does not affect other components of the model.

---

W4: Detailed Comparison with CoFIMA

We thank the reviewer for raising this important comparison. In table below, we report the full LoRA-M versus CoFiMA across multiple benchmarks, showing that the performances are very close:

To further analyze, we conducted a CoFiMA + Perturbation experiment (i.e., applying our task-vector perturbation during training and using CoFiMA merging). The results show improved performance over plain CoFiMA, but still underperform our full LoRA-P&M method. This indicates that:

• Our second-stage theoretical derivation is based on the model merging perspective, suggesting that perturbation is generally beneficial for merging-based strategies. As shown in Figure 2, such perturbation indeed improves the flatness of the loss landscape around the merged model;
• Our closed-form optimal merging coefficient and the perturbation are mutually reinforcing, yielding the best result.

---

Q1: Whether the perturbation may slow down the convergence speed

We measured convergence speed in terms of loss decay per training step. As shown in the table below as a example, perturbation training (LoRA-P&M) converges at a comparable rate to standard LoRA:

The added regularization term leads to slightly more stable but not slower convergence, confirming the efficiency of the stochastic perturbation.

---

Q2: Whether the proposed perturbation strategy remains effective when integrated with other continual learning methods, such as SD-LoRA.

We explored this question by applying our perturbation scheme to SD-LoRA, a state-of-the-art LoRA-based continual learning method. Interestingly, we observed performance degradation when adding perturbation to SD-LoRA:

This suggests that our perturbation training is particularly effective for merging-based strategies like P&M or CoFiMA, where parameter interpolation plays a central role in preserving task knowledge.

In contrast, SD-LoRA involves dynamically optimizing task-specific vectors during training. Our perturbations, which are aligned with a fixed task vector direction, may interfere with SD-LoRA’s learning dynamics and thus reduce performance.

This further highlights that the benefit of perturbation is targeted toward model merging, and not necessarily applicable to all continual learning strategies.

---

Q3: Whether the proposed method would be more effective when applied on top of non-PEFT methods.

While Section 3.4 introduces LoRA primarily as a means to reduce memory overhead, we emphasize that LoRA is also better aligned with the goals of continual learning (CL) beyond efficiency. LoRA tends to cause less interference with previously learned tasks[1], allows for easier task-specific adaptation, and requires fewer training examples and parameters to achieve strong performance. In contrast, full fine-tuning often leads to greater forgetting and lacks modularity. Therefore, even when memory is not a limiting factor, LoRA remains a more suitable and effective paradigm for continual learning.

---

We again thank the reviewer for their helpful feedback. We will incorporate the clarifications, additional comparisons, and ablations in the revised version.

[1] Dan Biderman, et al. "LoRA Learns Less and Forgets Less." TMLR24.

Dear Reviewer AEtr,

Thank you for your follow‑up question and for the constructive feedback. Regarding your remaining question, we would like to provide the following clarification:

Our merging method is derived from minimizing the total loss across all tasks, yielding a theoretically near‑optimal merging coefficient that requires no additional hyperparameters. Since computing the Hessian exactly is prohibitively expensive, we approximate it using the diagonal of the empirical Fisher information matrix, which inevitably introduces estimation errors [1]. This limitation has been discussed in the Limitation section of our paper. As a result, as shown in Fig. 2, the coefficient obtained by LoRA‑M via Eq. 8 does not always lie at the true loss minimum.

In contrast, CoFiMA guides model merging by introducing a hyperparameter $\lambda$, which leaves room for tuning and requires additional hyperparameter adjustment. The ablation study in the Fig. 4 of the original CoFiMA paper also shows that its performance is relatively sensitive to the hyperparameter $\lambda$. Fig. 2 also shows that, due to the inherent robustness of the pretrained model, a region of $\alpha$ values yields similar loss, so CoFiMA with a tunable hyperparameter can also achieve good performance. Moreover, as noted in our earlier rebuttal content, LoRA‑M demonstrates better compatibility with perturbation‑based training compared to CoFiMA.

We thank you again for your valuable feedback, and we hope our explanation addresses your concern. We would be glad to discuss further if you have any additional questions.

[1] Gido M. van de Ven. "On the Computation of the Fisher Information in Continual Learning." arXiv preprint arXiv:2502.11756.

Best regards,

Authors

We sincerely thank the reviewer for their valuable time and insightful feedback. Below we address the raised concerns point-by-point.

---

W1 & Q1: Whether our two-stage pipeline offers clear advantages over traditional one-stage training and further justification of its necessity and effectiveness.

We would like to clarify that our two-stage framework aims at introducing the model merging inference paradigm into continual learning (CL) to better preserve prior task knowledge.

Our motivation stems from a key observation: in traditional CL methods, the model trained on the current task is directly used to perform inference on both new and old tasks. This often leads to severe forgetting of previously acquired knowledge. In contrast, our method introduces a theoretically grounded post-training merging step. Specifically, we compute a convex combination of the previous inference model $\theta_{t-1}$ and the current task-specific optimum, where the merging coefficient $\alpha_t^*$ is derived in closed form (see Eq. (8)) to explicitly minimize the total loss increase across all seen tasks. Importantly, traditional CL approaches are equivalent to setting $\alpha = 1$ in our formulation—thus our method generalizes and improves upon one-stage training under the merging paradigm.

Regarding the reviewer’s concern on potential error propagation due to a multi-step process, we emphasize that our framework does not introduce additional backward passes. Structurally, our "Train + Merge Inference" pipeline is equivalent to standard "Train + Inference" approaches. The only difference is that we apply a theory-informed model merging step before inference (i.e., LoRA-M), which empirically reduces interference from new tasks and thereby mitigates error propagation. As shown in Figure 1 and Table 4, this leads to significant reduction in forgetting while preserving new task plasticity.

---

W2: The use of a uniform merging coefficient α across all parameters may be too restrictive.

Using a more fine-grained merging strategy for model parameters may potentially lead to a higher performance upper bound. However, such strategies would introduce additional computational overhead and theoretical complexity. Our experimental results also demonstrate that a unified merging coefficient already achieves strong and competitive performance.

We ran experiments comparing the global α with a more granular strategy—learning separate α values for each LoRA module. The results are as follows:

The performance differences are negligible. Moreover, we observed that the learned per-module α values were very similar to each other, suggesting that a global α is sufficient to capture the merging dynamics. Therefore, we adopt a single-α approach for its simplicity, efficiency, and theoretical tractability.

Based on prior research[1][2], a global α preserves the structural integrity of the task vector $\Delta \theta_t$ , which represents a coherent direction in parameter space. A parameter-wise α may introduce inconsistent scaling, distorting the internal correlations of $\Delta \theta^*_t$ and complicating the derivation of a closed-form solution.

---

W3 & Q2: The theoretical connection between the two stages of our framework.

There is a direct theoretical link: after deriving the optimal $\alpha_t$, we observed that even optimal merging leads to a total performance drop across tasks, quantified as $\sum_{i=1}^{t} \delta_i(\alpha_t)$ in Eq. 11. The Train with Perturbation step is specifically designed to reduce this upper bound during training.

The perturbation is not simply random noise; rather, it is a directional regularization based on $\Delta \theta_t$, approximating the Hessian-based quadratic penalty. This encourages convergence to flatter minima that are more robust to parameter interpolation, thus improving the final merged model’s generalization. This effect is clearly visualized in Figure 2, where the merged models from P&M exhibit significantly flatter and wider low-loss regions compared to the baseline.

---

W4 & Q3: Whether our theoretical assumptions are empirically grounded and suggests including parameter statistics to support their validity.

We would like to clarify that the assumptions we adopt—such as approximating the Hessian with the empirical Fisher Information Matrix, applying a second-order Taylor expansion of the loss, and neglecting higher-order terms—are not unique to our work. These are widely used and well-accepted approximations in the continual learning [3][4][5].

Such approximations have been extensively validated and empirically supported in recent deep learning theory and algorithms. Our use of them follows established practices rather than introducing new or unverified assumptions.

---

We hope that our clarifications and empirical justifications effectively address the reviewer’s concerns.

---

[1] Gabriel Ilharco, et al. "Editing Models with Task Arithmetic." ICLR23.

[2] Hongkang Li, et al. "When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers." ICLR25.

[3] James Kirkpatrick, et al. "Overcoming Catastrophic Forgetting in Neural Networks." 2017.

[4] Zhenyi Wang, et al. "A UNIFIED AND GENERAL FRAMEWORK FOR CONTINUAL LEARNING." ICLR24.

[5] Mei Li, et al. "BECAME: BayEsian Continual Learning with Adaptive Model MErging." ICML25.