We appreciate your time and highlighting these points. Below, we address each concern in detail.

Reference

We acknowledge the significance of AdaMerging[1] in model merging and promise to cite it to recognize its contribution. Additionally, we will expand the appendix to provide an comprehensive discussion on model merging.

After a careful review of the AdaMerging paper, we identify several key distinctions between their approach and ours. While both methods address adaptive model merging, AdaMerging operates within a multi-task learning framework where tasks are learned concurrently. In contrast, our work focuses on continual learning, where tasks are learned sequentially. Our merging strategy is specifically designed to consider the temporal dependencies between tasks.

Moreover, AdaMerging iteratively optimizes the merging coefficient at test time using entropy minimization on unlabeled data. In our approach, the coefficient is determined during training by a closed-form solution.

W1

While our two-stage training approach introduces additional computational costs, our results show that the overall training time remains well below double that of baselines, and the performance improvements justify the extra overhead. For instance, as shown in Table 4, GPM attains 63.90% ACC in 434.66s, whereas our method achieves 67.62% in 584.14s. The second stage adds less than 35% to the duration of the first stage, making our approach highly competitive compared to single-stage methods like TRGP, which requires considerably more training time (776.72s) due to complex operations.

The reasons for this efficiency are as follows:

• The unconstrained training in second stage incurs a lower per-epoch cost (0.52s) than gradient projection (0.85s) and typically converges in fewer epochs when early stopping is applied.
• In contrast to the gradient projection, which spends additional time computing the subspace for each task, the second stage avoids this overhead entirely.

W2

We would like to highlight that our method does not significantly increase storage requirements.

Below, we first present the experimental results and then analyze why the BECAME framework is memory-efficient.

To provide a detailed comparison, we divide the total memory into three parts: model, the additional memory for variables needed for training future tasks and the temporary memory required only during training. We further divide the additional memory based on training stage for analysis. The results are as follows:

• The second training stage introduces only a modest storage overhead. As shown in the 5th and 6th rows of the table, the only additional memory for the second training stage is the precision matrix, which is comparable in size to the model parameters and determined solely by the model architecture rather than by the baseline method.
• The increment of the additional memory in the first training stage is a direct result of improved overall performance, which follows the inherent feature of the gradient projection methods. After performing our model merging, the model's capability of feature extraction is enhanced so that the dimension of output feature space expand and more base vectors need to be stored for future task training.
• The memory overhead of our method is competitive to GPCNS and TRGP, which required more storage than our method even they only have one training stage.

We hope these detailed responses address your concerns and further enhance the robustness of our approach.

We appreciate your careful review and the constructive suggestions. Your recognition of this work is truly encouraging. Here we present our detailed clarifications and updates, which we believe further enhance our paper.

Evaluation&C1

In response to your suggestion, we compute Averaged Anytime Accuracy (AAA) using the formula: $AAA = \sum_{i=1}^T AA_i$ with $AA_i = \frac{1}{i} \sum_{j=1}^{i} A_{i,j}$. The results are shown in the tables below:

These results confirm that our method continues to yield substantial improvements when evaluated with AAA, with trends closely aligning with ACC. Due to space limit, we will include these results in the appendix.

Theory&C2

Thank you for your suggestion. We will revise the statement at lines 181–182 to clarify that the log prior remains constant only under a uniform prior. Additionally, we have verified that the parameter prior is predefined and that this adjustment does not affect subsequent derivations or results.

W1

The observed degradation in the Backward Transfer (BWT) metric in some cases reflects a deliberate trade-off favoring enhanced plasticity and overall performance.

• Sensitivity of BWT. BWT of task t is determined by its immediate post-training performance ($A_{t,t}$) and its final performance after learning all tasks ($A_{T,t}$). In some instances, a higher $A_{t,t}$ may lead to a lower BWT even when overall ACC improves.
• Stability-Plasticity trade-off. Our approach intentionally prioritizes increased plasticity to boost performance on new tasks, which may slightly reduce stability (as measured by BWT). However, the overall accuracy benefits from this trade-off.

We emphasize that our method does not compromise the performance of earlier tasks beyond an unacceptable level; rather, it strategically balances the stability-plasticity trade-off to achieve better overall performance.

Q1

Yes, our method is equally applicable to class-incremental learning (CIL). The theoretical framework and model merging mechanism are independent of task- or class-incremental learning. Since our approach operates directly on model parameters, it is effective in both scenarios. To validate this, we conducted a simple experiment by testing without task IDs, following the standard setup for CIL.

The results indicate that our method also improves accuracy in the CIL setting. The improvement may be attributed to the enhanced balance of performance across tasks achieved through merging (see Line 408). We are delighted to investigate this aspect more in future work.

We hope these detailed responses clarify our revisions and validate the robustness of our approach. We sincerely appreciate your valuable feedback.

We sincerely appreciate your thorough review and the insightful comments on our theoretical framework and experimental validations. Below, we detail our clarifications and additional proof.

Theory

1. We agree that the assumption of independence across tasks is necessary for Eq. 6 and will fix it, while Eq. 1 is derived by definition and imposes no such constraint.
2. Our proof of Lemma 3.1 utilizes fixed endpoints to show the existence of a merged model with a lower cumulative loss. Importantly, this proof does not assume that the loss function is globally convex with respect to the model parameters.

W1

We conduct additional ablation studies in response to your advice. And the results indicate our method consistently outperforms alternative merging strategies.

Please refer to the Experiment part of the response to reviewer Nkj6. We will include them in appendix to further support our method.

C1

We promise to include a detailed discussion on model merging in appendix in our revised vision, as well as classical literature on Gaussian mixture models and multivariate mixtures.

Q1

Thank you for the suggestion, here we supplement a theoretical analysis on why $\Delta\theta$ in Eq. 17 can be substituted by $\hat{\theta}_t-\theta_t^{GP}$.

The key point is to prove that Eq. 10 holds when $\theta_{t-1}^*$ is replaced by $\theta_t^{GP}$.

Given $\Lambda_{t-1}$​ is symmetric and semi-positive definite, as it is the Hessian of the negative log posterior, we have

$$(\theta-\theta_t^{GP})^{\top}\Lambda_{t-1}(\theta-\theta_t^{GP})=(\theta-\theta_{t-1}^*)^{\top} \Lambda_{t-1}(\theta-\theta_{t-1}^*)+2(\theta-\theta_{t-1}^*)^{\top}\Lambda_{t-1}(\theta_t^{GP}-\theta_{t-1}^*)+(\theta_t^{GP}-\theta_{t-1}^*)^{\top}\Lambda_{t-1}(\theta_t^{GP}-\theta_{t-1}^*).$$

Then we only need to prove $\Lambda_{t-1}(\theta_t^{GP}-\theta_{t-1}^*)=0 \ (1)$. We define $d=\theta_t^{GP}-\theta_{t-1}^*$.

$\Lambda_{t-1}$ can be decomposed as $\Lambda_{t-1}=Q^{\top}AQ$, as $A$ is a diagonal matrix of the eigenvalues.

To prove (1), we do a second-order Taylor expansion of $L_{1: t-1}(\theta_t^{GP})$ around $\theta_{t-1}^*$, yielding: $d\Lambda_{t-1}d=dQ^{\top}AQd=0$, since $L_{1: t-1}(\theta_t^{GP})\approx L_{1: t-1}(\theta_{1-t}^*)$ and $\theta_{t-1}^*$ is an optimum.

Given that $A_{ii}\geq 0$, it follows each element of $Q$ must be 0, leading to $\Lambda_{t-1}d=Q^{\top}AQd=0$​​​​.

Due to the character limit, we will provide a detailed proof in our revised version.

Q2

1. Our theory does not require Eq. 10 to be convex. We only claim that when merging model from$θ_{t−1}^*$ to $\hatθ_t$, the total loss in Eq.11 is convex with respect to $\lambda$ (Line 236).
2. We treat $\hatθ_t$ as a scalar for the following reasons:

• When $\lambda$ is scalar, our analysis allows us to readily show that $\tilde L_{1:t}(\lambda)$ is convex .
• From our proof in Q1, when $\lambda$ is a vector, $\Lambda_{t-1}\lambda(\theta_t^{GP}-\theta_{t-1}^*)$ may not be 0, preventing the replacement of $\theta_{t-1}^*$ with $\theta_t^{GP}$ in our BECAME framework.
• There exists a linear connector from $\theta_A$ to $\theta_B$ when they are trained sequentially [1], as also indicated in the right column of Line 146.
• Our ablation studies and Table 3 have demonstrated that per-parameter merging does not necessarily yield better results.

3. We also provide additional experiments and analysis to validate that optimizing Eq. 10 via model merging is more efficient than regularization.

Now we present the analysis to further elaborate:

• Adding regularization to the loss function slows down training. The additional time grows as the parameter dimension increases, whereas model merging without regularization leads to faster training.
• Regularization is added to prevent the loss of old tasks from increasing. For a specific $\theta_i$ in $\theta$, optimization for the new task follows the loss based on the network architecture, while for old tasks it is constrained by the second-order norms. This explains why the results of regularization differ from those of model merging, even with constrained direction. Our approach naturally avoids this conflict and consistently achieves the best results.
• Due to the aforementioned imbalance, weight of the regularization term should be carefully tuned to get optimal performance, while our model merging does not require such manual tuning.

[1] Linear mode connectivity and the lottery ticket hypothesis. ICML 2020

We appreciate your time for the review and respect your concern. However, we respectively argue our current experiments have sufficiently validated our contributions.

Experiment, W2

We believe our experimental design and results sufficiently support our claims for the following reasons:

1. We have already compared various merging methods within the BECAME framework in Table 3, as discussed in Line 366. Among these methods, CoFiMA is a per-parameter merging method based on Fisher Merging. The results show that our method outperforms alternative merging strategies in both ACC and BWT. This empirically supports the superiority of our merging coefficient and the adaptability of model merging in gradient projection (GP) methods. If there are any other baselines you would like us to compare, we are happy to include them.
2. The BECAME framework itself is a key contribution alongside the merging method, making its validation through experiments essential.
3. One of our primary motivations is to enhance the adaptability of existing GP methods, and the results in Tables 1 and 2 show that BACAME effectively integrates with various GP methods, significantly improving their performance. To further strengthen our study, we conduct experiments on applying model merging directly to CL without GP:

These results further highlight the effectiveness of our merging coefficient compared to other methods.

Claims&Theory&W3&Q2

We respectfully argue that our theoretical analysis using Laplace approximation is valid for the following reasons:

1. Given the inherent complexity of deep learning optimization, which is influenced by numerous factors, it is impractical to analyze every possible case explicitly. Therefore, approximation techniques are not only common but also essential for deriving meaningful theoretical insights**.** This is widely recognized in the field of machine learning.
2. The use of Laplace approximation in neural networks has been well-established since 1992 (A Practical Bayesian Framework for Backpropagation Networks, cited by 4264), and numerous studies have built upon this foundational framework. Recent works (Lee et al., 2017; Marouf et al., 2024; Kirkpatrick et al., 2017; Ritter et al., 2018) have successfully applied Laplace approximation in CL, further validating its applicability
3. Model merging involves two key components: the model parameters {$\theta\_i$}$_n$ and their corresponding weights {$\lambda_i$}$_n$. In our setting we have {$\theta_i$}$\_n$={$\theta\_{t−1}^*,\theta_t$}, and our optimal merging coefficient is derived based on the merging trajectory between these two endpoints.
4. We have provided additional empirical evidence through loss function visualization (Figs. 1 and 2) and comparative experiments (Table 3), further supporting the reliability of our theoretical analysis.

Relation&W1&Q1

1. Both Fisher Merging and our method are based on Laplace approximation and use Fisher for calculation, yet they differ significantly in many aspects. Below is a detailed comparison:

Our method fills a crucial gap in existing merging approaches for CL by considering the influence of prior tasks on new task learning. 2. Using a scalar $\lambda$ rather than a vector is a deliberate choice for theoretical analysis. Ablation study and Table 3 also indicate that per-parameter merging does not necessarily yield better results. Please refer to Q2 of the response to reviewer H5hN for more details.

We hope these detailed responses and additional experiments address your concerns.