Wall-clock time

Lacks wall-clock time comparison ($\mathcal{O}(1)$ does not shed any light on the actual cost incurred), for instance that used to update the FIM, which is important to faithfully evaluate the method w.r.t. to other works.

In response to the reviewer's suggestion, we have included a wall-clock time analysis in Sec. H.6 of the Appendix. Specifically, considering Split ImageNet-R, Split RESISC45, and Split CropDisease, we measured the per-task runtime of ITA and IEL (both trained with full fine-tuning), as well as that of three existing approaches to incremental learning: DER++ (rehearsal), TMC (compositionality), and SEED (ensemble learning). Notably, the runtime includes the time required to setup the regularization of ITA and IEL (e.g., the computation of the FIM statistics).

Among the methods compared, SEED appears to be the most efficient. However, the runtimes of both ITA and IEL are comparable to, or even better than, those of baseline methods like DER++ and TMC. Specifically, on ImageNet-R and RESISC45, the average runtime of ITA/IEL is approximately $\frac{2}{3}$ that of DER++.

We attribute the reasonable training times of IEL/ITA to the efficient procedure we employed for estimating the FIM, which leverages batch-wise parallelization (refer to Sec. E of the Appendix, see also George, 2021), combined with the use of closed-form gradient computation (refer to Sec. C of the Appendix).

Comparison

The need for the task pre-consolidation (linear probing, etc.) stage requires additional training steps, so comparisons to most existing methods that do not require this would not be completely fair. If the paper did not implement control steps to make fairer comparisons, then this should be mentioned explicitly in the limitations.

We understand the point raised by the reviewer. The comparisons in Tab. 1 were primarily designed to position our approaches within the leaderboard of continual learning. Hence, we selected a heterogeneous and challenging snapshot of state-of-the-art approaches, ranging from older rehearsal-based methods to more recent prompt-tuning techniques. Compared to these methods, we acknowledge that our approach introduces additional overhead due to the pre-tuning stage. On the other hand: i) we avoid the double forward pass required by prompt-tuning approaches such as L2P and CODA-Prompt; ii) we employ a rehearsal-free approach that does not require maintaining a memory buffer (as in DER++); iii) we avoid the additional overhead introduced by TMC in computing the JVP.

Hence, the fairest apple-to-apple comparison among all methods is somewhat elusive due to their heterogeneity. If the reviewer deems it appropriate, we can include this discussion in the supplementary materials.

Questions

I see some replay-based methods like DER++ are included in Table 1, but I don't see any mention of the experimental setup used for these comparisons (e.g. assumptions on buffer size).

We sincerely apologize to the reviewer for this oversight. In the updated version of the main paper, we have explicitly stated the buffer size (see the caption of Tab. 1). Specifically, DER++ utilizes a buffer of $1,000$ elements across all the experiments we conducted.

Equation (13) suggests that by minimizing $\Omega$, IEL also seeks to encourage linearity in the loss, as done in ITA. Can you provide some intuition (formal or informal) regarding under what regimes IEL fairs better than ITA, and vice-versa?

This point is very appealing and aligns with our interests for future work. While ITA trains each model separately, IEL directly trains the ensemble as a whole, rather than each task in isolation. This implies that, with IEL, each task can benefit from the knowledge already accumulated within the ensemble, potentially enabling improved positive transfer. However, to substantiate this claim, we would need additional experimental analyses beyond those reported in this paper, such as comparing the two methods in scenarios where knowledge overlap between consecutive tasks is possible. Hence, we leave this investigation for future work.

We thank the reviewer for their careful review and helpful suggestions. We have provided thorough responses to each of the points raised.

Sec 2.2 and second-order analysis

The regularization condition proposed in Sec 2.2 does not seem to necessitate the second-order analysis, since it simply aims to enforce a local (first-order) linear structure by making perturbation smaller.

We would like to clarify that the regularization term in Sec. 2.2 actually stems from a second-order analysis. Indeed, the regularization objective based on the FIM originates from a second-order approximation of the KL divergence between the output distribution of the current model and that of the pre-trained model. The full derivation was omitted as it builds on well-established proofs $a, b$ and does not constitute part of our original contributions. We referenced this connection in line 172 of the original submission, and more theoretical details are provided in the first section of the Appendix in $a$ .

$a$ Chaudhry, Arslan, et al. "Riemannian walk for incremental learning: Understanding forgetting and intransigence." ECCV. 2018.

$b$ Schwarz, Jonathan, et al. "Progress & compress: A scalable framework for continual learning." ICML. 2018.

Wording

The wording on "incrementally train(ing) the whole composed model rather than its individual components" used throughout the paper is misleading ... I would suggest modifying the terminology for better clarity.

Based on the comment of the reviewer, we thoroughly analyzed the original submission and identified the following critical lines:

• line 70: ...optimizing each model individually vs. optimizing the whole composed model directly
• line 106-107: ... can we incrementally train the whole composed model rather than its individual components (ensemble training)
• line 213: For these reasons, we conduct a theoretical analysis of the dual perspective: the incremental training of the composed model (ensemble training).

To clarify the terminology, we have implemented the following modifications:

• line 71 ... optimizing the loss of each model individually vs. optimizing the loss of the composed model
• line 106 ... Instead of optimizing each model on its individual loss, could we optimize each model based on the loss of the whole composed model (ensemble training)?
• line 211-212: ... For these reasons, we adopt the dual perspective, in which each model is directly optimized using the loss of the composed model (ensemble training).

Optimality assumption

L120 assumes that the pre-trained initialization $\theta\_0$ is globally optimal for a quadratic model trained on the union of all tasks. This seems to be a very strong assumption that likely would never hold in practice (why train/compose when the initialization is already optimal?), contrary to what is claimed in footnote 1. This weakens the theoretical analysis.

We acknowledge the optimality of the base model represents a strong assumption that may not hold in practice. In the future, we are committed to extending this work by exploring more relaxed conditions that may yield equivalent theoretical results.

That being said, we believe that incremental training remains valuable even in a scenario where the base model is already considered "locally optimal" with respect to all tasks. Indeed, our framework offers a practical mechanism to decompose the model into multiple weight perturbations, each tailored to specialize in a specific task distribution and potentially achieve a better local minimum. By doing so, the task vectors derived from fine-tuning the base model would enable both specialization and unlearning capabilities, as demonstrated in our experiments.

Moreover, while the assumption of "optimality" for the base model may be considered unrealistic, it serves as a theoretical ground to understand why model merging techniques have been shown to perform more effectively when starting from stronger base models --- e.g., those that are wider or augmented through similar forms of pre-tuning (see $a$ , a work referenced by Reviewer tsHk).

We have reformulated footnote 1 to include some important references provided by Reviewer tsHk.

$a$ Yadav, P., Vu, T., Lai, J., Chronopoulou, A., Faruqui, M., Bansal, M., & Munkhdalai, T. (2024). What Matters for Model Merging at Scale?. arXiv preprint arXiv:2410.03617.

We are grateful to the reviewer for their constructive feedback and valuable insights. Please find our detailed responses to each comment below.

Connection with the NTK

The motivation for using the second-order linear approximation is unclear. To my understanding, this work employs a similar approach to neural tangent kernel (NTK) analyses. However, in the linear approximation, NTK typically uses the first-order Taylor expansion, here the author employs the second order. ... So, it would be great the author add a section and comparing their work with $1-2$ , especially the Eq. 3-4 with Lemma 3.1 in $2$ .

Despite the common use of Taylor approximations, we highlight that there are some notable differences between our approach and those leveraging NTK analysis:

• Our approach employs the second-order Taylor approximation of the loss function, aligning with the spirit of works such as Mirzadeh et al. (2020) and Chaudhry et al. (2018). Our focus is on the geometry of the loss landscape and understanding how the loss changes (i.e., potential performance degradation) with respect to weight perturbations.
• To the best of our knowledge, works dealing with NTK analyses typically apply the first-order Taylor approximation to the output space, i.e., the activations of the final layer. Their approach primarily investigates the learning dynamics of the output, whereas our method focuses on approximating the loss function.

We believe that these two frameworks are not mutually exclusive, as they serve different aims. Consequently, they could potentially be integrated to derive alternative solutions. We leave this intriguing, albeit complex, exploration for future work.

Why not use the first-order Taylor expansion or perhaps a third-order expansion?

With the NTK analyses, the output of the model can be approximated during training by its first-order Taylor expansion around its initialization. However, this approximation has often been shown to be invalid at finite widths, with the training dynamics not being adequately captured by the first-order approximation. Indeed, in their 2024 study, Ortiz-Jimenez et al. observed that models like CLIP do not fine-tune within a linear regime and lack kernel behavior. To address this, both Ortiz-Jimenez et al. (2024) and TMC by Liu & Soatto (2023) proposed training linearized versions of these models directly, resulting in more disentangled weights.

Instead, we consider the Taylor expansion of the loss function and include a second-order term, with the hope that this could lead to a better approximation. Hence, our approach is not explicitly grounded in NTK or kernel-based behavior. Although it could be extended to even higher-order terms, we limit our focus to the second order due to practical constraints, such as the challenges associated with estimating the third-order derivative of a deep neural network.

NTK and Continual Learning

Also, there are NTK-based class incremental learning works $3-5$ . The authors could include a section in the introduction or discussion to clearly point out the differences and benefits of using their approach compared to these works.

We thank the reviewer for highlighting these references. We have updated Sec. 4 (Relation to Existing Work, see ll. 369-377) to explicitly address NTK-based approaches for class-incremental learning.

However, we would like to point out that the original submission already includes a comparison with Tangent Model Composition (TMC) by Liu & Soatto (2023). This work leverages model linearization to address the shortcomings of standard non-linear fine-tuning, which has been shown to not present kernel behaviour. In this respect, model linearization is essentially equivalent as training a kernel predictor with neural tangent kernel defined as $k\_{\rm NTK}(x,x') = \nabla\_{\theta} f(x;\theta\_0)^\top \nabla\_{\theta} f(x';\theta\_0)$. In this respect, we established a theoretical connection between two inequalities (see Eq. 6 and Eq. 7), as well as an experimental comparison with TMC (see Tab. 1 and Tab. 3). Specifically, we demonstrate that our method significantly enhances the task arithmetic capabilities of the merged model, outperforming NTK-based methods such as TMC.

We appreciate the reviewer's insightful feedback and suggestions. Below, we provide a comprehensive response to each of the comments.

Optimality of pre-training model

... That being said, I find that approximating this condition by regularizing the task vectors to stay close to the base model somewhat of a big conceptual leap. By construction, the base model performs sub-optimally on the tasks for which we optimize a task vector, otherwise the task-specific optimization does not serve much purpose. Therefore, in Eq. 5, wouldn't replacing ${\theta}\_{t}$ with ${\theta}\_0$ for any $x \sim p_{t'}, t' \neq t$, still make the upper bound loose, and therefore not useful?

We understand the concern expressed by the reviewer and agree that, by construction, the base model likely performs sub-optimally on the downstream tasks. As rightly pointed out, replacing $\theta\_t$ with $\theta\_0$ in Eq. 5 would indeed loosen the upper bound, but not to the same extent as the right-hand side of Eq. 5. As shown in Fig. 1 of the main paper, the upper bound yielded by the base model $\theta\_0$ (yellow line) is consistently better than the original upper bound in Eq. 5 (without regularization, see dark gray rectangles). This occurs because the base model remains frozen and, by construction, maintains broader capabilities (although not optimal) across all tasks. These capabilities are instead lost when training models individually without regularization, hence rendering the upper bound in Eq. 5 excessively loose.

The core intuition, therefore, is that it is better to regularize using a (although not optimal) base model rather than leaving individual models to move arbitrarily in parameter space.

ITA and decentralized learning

... given that the FIM is updated sequentially, how would one train several tasks in a decentralized manner without communicating FIM statistics?

We would like to discuss further our original statement. In our scenario, the factor that either enables (ITA) or prevents (EWC) decentralized learning is the choice of weight distribution used for regularization. Specifically, while EWC’s FIM pertains to the previous task, our approach computes the FIM statistics with respect to the base model. In our opinion, this approach allows for a decentralized learning setup, which can be summarized as follows:

1. Multiple disjoint tasks are distributed across several ITA clients, with an ITA server responsible for handling communications.
2. Each ITA client performs pre-tuning and computes its local FIM for its training task.
3. Each ITA client communicates its FIM to the ITA server.
4. The ITA server merges the local FIMs and returns the result to the clients, as is common in Federated Averaging $a$ .
5. Each client learns its task with additional regularization provided by the merged FIM.

In this way, multiple task vectors can be trained on different tasks in parallel; the only requirement is to set up a communication phase where clients send their FIMs to the server node. In contrast, an EWC-like term, which regularizes each task based on its predecessor, would require training the multiple task vectors (or clients) in sequence.

$a$ McMahan, B., Moore, E., Ramage, D., Hampson, S., & y Arcas, B. A. (2017, April). Communication-efficient learning of deep networks from decentralized data. AISTATS.

Question: ITA vs IEL and differences in their task vectors

... How different are the task vectors learned with ITA vs IEL? Do you find them to be well-aligned?

Thank you for the suggestion. We have included an additional evaluation in Sec. H.5 of the Appendix, where we compute the cosine similarity between the task vectors learned with ITA and IEL across each layer of the ViT-B/16 backbone. For a complete analysis, we evaluate both the average cosine similarity between individual task vectors $\tau\_t^{\text{ITA}}$ and $\tau\_t^{\text{IEL}}$, and the cosine similarity between the composed task vectors $\tau\_{\mathcal{P}}^{\text{ITA}}$ and $\tau\_{\mathcal{P}}^{\text{IEL}}$.

Our analysis indicates that the task vectors are largely similar, particularly for datasets that require significant adaptation. This is mostly evident in out-of-distribution datasets ( i.e., RESISC45 and CropDisease) as well as in low-resolution ones (i.e., CIFAR-100).

However, when focusing on the similarity between individual task vectors we observe minimal alignment for CUB-200 and MIT67. We attribute this to the higher complexity of these datasets, with CUB-200 involving fine-grained classification and MIT67 involving capturing nuanced contextual differences, such as those between $`art studio`$ and $`classroom`$.

Regardless of this distinction, we observe that the similarity of the composed task vectors consistently exceeds the (average) similarity between individual task vectors.

Thank you for your thoughtful feedback and recommendations. We address each point in detail in our responses below.

Connection to related works

These works are not cited in the current version, maybe because these connections were not obvious to the authors. I suggest the authors revisit this potential connection and cite the related works.

We sincerely thank the reviewer for their contribution in connecting our work to other fields and highlighting some of the most recent, closely related studies. We have incorporated all five works mentioned in their summary throughout the paper.

Given its importance, we had already included a section discussing related works on model merging in the original submission. However, as our work spans multiple fields and explores their potential intersections, it is challenging to comprehensively address all relevant literature while adhering to the page limit set by ICLR. For this reason, that section was moved to the supplementary materials before submitting the paper, specifically to Sec. F.

Questions and clarifications

... it must be acknowledged that modular architectures, such as MoE have become increasingly widely used in recent years (maybe as wide as monolithic models).

We agree with this observation and have rephrased the first paragraph of the Introduction to more accurately reflect the feedback (see ll. 028-032).

ll. 152-157: the paragraph seems to argue that the right-hand side of equation 5 will increase as new tasks are learned. But in equation 5's right-hand side, each individual loss starts from the base model and does not depend on any form of continual learning. So I am not sure I understand the argument being made here.

On the right-hand side of Eq. 5, we observe a sum of the empirical risks achieved by each individual model across all tasks. Since each model is trained on only one of these tasks, the upper bound is likely to increase when the evaluation is expanded to include a new task -- particularly if the new task is poorly aligned with the training distributions learned by the individual models in previous tasks.

If any part of this explanation remains unclear, please do not hesitate to contact us for further clarification.

ll. 152-157: would the pre-consolidation step be necessary in language modelling, where the output space is usually shared across tasks?

In language modeling, where tasks generally share an output space, a pre-consolidation step may not be strictly necessary. However, we believe that strategies to enhance the base model, in the same spirit as pre-tuning, can still be advantageous. For instance, domain adaptation could be employed to strengthen the base model, possibly by leveraging a reference unlabeled dataset with semantic similarities to the downstream tasks. Alternatively, data from the first task could be used for this purpose, following an approach similar to "First Session Adaptation" proposed by $a$ . We intend to explore these strategies, as well as tasks involving CLIP and Large Language Models (LLMs), in future work.

$a$ Panos, A., Kobe, Y., Reino, D. O., Aljundi, R., & Turner, R. E. (2023). First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning. ICCV.