Efficient Model Editing with Task-Localized Sparse Fine-tuning

We thank the Reviewer for their constructive feedback on our work. We appreciate the positive assessment, particularly regarding the thorough testing of our claims, the comprehensiveness of our analysis, and the quality of the writing. We respond below to the weaknesses and questions raised in their Review.

• (W1) Linear behavior + Parameter overlap across tasks. This question has two parts. Regarding the first part, “linearized behavior” refers to the fact that the output of the model $f(x,\theta_0)$ is well approximated by its first-order Taylor approximation centered around $\theta_0$. Equation 9 in the paper formalizes this concept mathematically. We report it here for convenience. As $\||\theta\_t - \theta\_0\||^2 = \||\tau\_t\||^2 \rightarrow 0$,
  $f(x,\theta_0 + \tau_t) = f(x,\theta_0) + \tau_t^\top \nabla_\theta f(x,\theta_0) + \mathcal{O}(\||\tau_t\||^2)$
  The quality of the approximation is thus tied to the norm $\||\tau_t\||^2$: the smaller, the better. Updating only a subset of parameters by masking some gradients (sparse fine-tuning, as done in TaLoS) instead of all the parameters (full fine-tuning), will lead to a smaller norm $\||\tau_t\||^2$, provided that the gradient updates are similar in magnitude for the unmasked parameters between the two fine-tuning paradigms. Intuitively, most of the elements of the weight vector $\theta_0$ will remain fixed during fine-tuning, resulting in many entries in the task vector $\tau_t = \theta_t - \theta_0$ to be zero. Hence, the smaller the norm, the more accurate the approximation. We added a clarification on this aspect at L270.
  Furthermore, by following [1] we provided empirical evidence of this fact, as discussed at LL1132-1148. In Figure 6 of Appendix A.4 each red marker’s y-coordinate is the single-task fine-tuning accuracy of models obtained by TaLoS while the x-coordinate is the single-task fine-tuning accuracy achieved by linearizing the models obtained by TaLoS. In most cases, adding an explicit linearization to the models obtained by TaLoS doesn’t modify the accuracy, as the markers fall on the bisector (black dashed line). This indicates that the fine-tuning via TaLoS, indeed, exhibited linearized behavior.
  In summary, TaLoS achieves linearized behavior by sparsely updating a small subset of parameters that exhibit the lowest sensitivity to a task (measured by $\mathbb{E}\_{x \in \mathcal{D}\_t} [|\nabla\_\theta f(x,\theta\_0)|]$) and keeping fixed the remaining ones.
  Regarding the second part of the question, we assess in Figure 11 (Appendix A.7) the parameter overlap of the shared subset of low-sensitivity parameters across task vectors identified by TaLoS at 90% sparsity. For each parameter, we compute the mean Intersection over Union (mIoU) of masks, between each task pair: starting from pre-trained parameters $\theta_0$, we predict the mask on task $t$ and then check its intersection over union against the mask predicted on task $t’$ (which acts as a ground truth). A mIoU of 1 signals perfect mask overlap between tasks. The number of parameters selected by each mask is 10%, in line with the experiment of Figure 1. Smaller vision models (ViT-B/32) exhibit high parameter sharing (>0.7 mIoU) of low-sensitivity parameters, while smaller language models (T5-Small) share fewer of them (0.3–0.5 mIoU). Larger models in both vision and language domains share more low-sensitivity parameters across tasks.

Thank you for the question. Exactly. Task-wise AdaMerging seeks to refine task addition by learning a unique merging coefficient, $\alpha_t$, for each task vector $\tau_t$. This approach augments the standard practice, where a single $\alpha$ is determined at validation time for all task vectors [1, 2, 3]. As noted in [2], tuning individual merging coefficients does result in mildly higher performance. However, it comes at the cost of significantly more computation, with relatively modest gains. A variant, Layer-wise AdaMerging, further diversifies this by assigning a separate merging coefficient for each layer rather than a global coefficient per task vector. Both AdaMerging approaches introduce additional degrees of freedom, aligning with the logic of TaLoS and generally translating to higher performance.

Regarding TIES-Merging, its sign conflict resolution mechanism relies on a heuristic rule applied on $\tau_t$, which may increase the dot product $|\tau\_t^\top \nabla\_\theta f(\cdot,\theta\_0)|$: a quantity that TaLoS seeks to keep minimal, as a dot product closer to zero across $t' \neq t$ tasks indicates that the task vector $\tau_t$ obtained via TaLoS is performing more localized edits primarily in its own task input space with support $\mathcal{D}\_t$ (for more details we point to Equation 6 and discussion at LL182-204).

To investigate empirically whether the sign resolution of TIES is interfering with the logic of TaLoS, we compute $|\tau\_t^\top \nabla\_\theta f(\cdot,\theta\_0)|$ across all tasks $t’=1,..., T$ for both TaLoS and TaLoS + TIES-Merging. For reference, we also report the results for LoTA, with all methods at 90% sparsity. For vision we report ViT-B/32 fine-tuned on RESISC45 and for language we report T5-Small fine-tuned on QASC as no variations in the trend were observed on all the other tasks.

On ViT-B/32 (RESISC45), TaLoS + TIES-Merging exhibits slightly higher $\tau_t^\top \nabla_\theta f(\cdot,\theta_0)$ values compared to TaLoS, correlating with marginally reduced merging performance (as discussed previously). On language tasks using T5-Small (QASC), the impact of applying TIES-Merging to TaLoS yields significantly higher $\tau_t^\top \nabla_\theta f(\cdot,\theta_0)$, aligning TaLoS + TIES-Merging to LoTA.

References:

[1] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. "Task arithmetic in the tangent space: Improved editing of pre-trained models." Advances in Neural Information Processing Systems 36 (2023).

[2] Ilharco, Gabriel, et al. "Editing models with task arithmetic." International Conference on Learning Representations (2023).

[3] Tang, Anke, et al. "Parameter efficient multi-task model fusion with partial linearization." International Conference on Learning Representations (2024).

Dear Reviewer,

as the end of the discussion period is approaching, we kindly ask if our responses have adequately addressed their concerns or if there are any additional questions we can clarify.

We thank the reviewer for recognizing the novelty of our work and the comprehensiveness of our empirical validation. We address each point raised by the reviewer below:

• (W1 + W2) Improving the clarity of the paper & general fixes. We updated the caption of Figure 1 to be more clear about the chosen pruning criterion and the sparsity ratio. Regarding the quoted sentence ("To prevent interference between tasks and enable task arithmetic, prevents significant changes to the parameters that are highly sensitive to multiple tasks."), it does not appear in our manuscript, so we are unable to address this concern. Please see our response to Question 1 (Q1), for clarifications about our approach and the algorithm.

• (W5 + Q4) Relationship between gradients & task interference. We never claimed in our paper to draw any “direct relationship between gradient values and task interference”, so we ask the reviewer to be more specific about this point to be able to address their concern.

• (W3 + Q2) Theoretical guarantees or convergence properties. Our submission dedicates significant space to the theoretical foundation of our method. We refer the reviewer to Sections 3 and 4 for more details. However, our main focus is the empirical evaluation of model merging algorithms for deep learning models (e.g. CLIP and T5) pre-trained and fine-tuned with stochastic gradient descent, which are nonlinear, non-convex settings. Deriving convergence guarantees for these settings is an interesting research direction, but out of the scope of our work.

• (Q1) Clarifications about TaLoS. Here we re-state for major clarity our TaLoS approach. The pseudocode of TaLoS was provided in Appendix A.2. We made sure to point to it clearly in the main paper (now in L247 of the revised version).
  First of all, $x$ refers to a single example. Mask calibration consists of estimating which parameters to sparsely update at fine-tuning time. The calibration runs only once per task before fine-tuning the pre-trained model parameters $\theta_0$.
  As reported at LL239-249 and Algorithm 1, we construct the gradient mask $c \in \mathbb{R}^m$ by setting to 1 the entries in $c$ corresponding to the average output gradient values $s = \mathbb{E}\_{x \in \mathcal{D}\_t} [|\nabla\_\theta f(x,\theta\_0)|] \in \mathbb{R}^m$ that are below the threshold $s_k$. The remaining entries in $c$ are set to 0.
  The threshold is selected as the $k$-th largest entry in the vector $s$, where $k$ is a user-defined hyperparameter and controls the amount of parameters that will be updated during sparse fine-tuning. $s$ can be obtained efficiently using only a small subset of data of the task by computing per-example gradients.
  The data used in this process is the validation split of the considered task. As detailed in Appendix A.1 at LL894-899, this choice aligns with the setting adopted by LoTA for mask calibration, ensuring a fair and direct comparison between our calibration method and theirs.
  We remark that the focus is on fine-tuning pre-trained models to produce high-quality task vectors $\tau_t$ that effectively capture the knowledge of every $t$-th downstream task. We can access one task at a time when fine-tuning i.e., we are constrained to individual task training. The data from both the training and validation splits is available in our setting as prescribed by common task arithmetic practices [1, 2].
  As explained in Appendix A.2 (LL1016-1025, and now also at LL245-248 of the main submission), the mask $c$ could be calibrated by performing a single pass on the data split used i.e., in a single round. However, noisy gradients may affect this process, as evidenced in the Pruning-at-Initialization (PaI) literature [3]. A general solution adopted there consists of running multiple rounds and keeping only the bottom-$p$ parameters (where $p \geq k$) at each round, excluding the rest from mask calibration. Thus, we followed that strategy with an exponential decay of the current sparsity $p$. We repeat this process until we reach the user-defined target sparsity $k$, obtaining the final mask $c$.
  We hope the algorithm of our approach is clearer now. Should you have any more specific concerns, please let us know and we will be happy to answer. We also encourage the reviewer to read the responses to other reviewers for further clarification.

Thanks for the reply. I'd like to re-state my comments point to point.

1. Grammar errors in line 229, '...update the only the ones where...', which is also the same sentence in my original comment though there's a typo when pasting the sentence.

2. Thanks for pointing it out, since the sparsity param is very important, I suggest it should be in the main text.

3. After carefully rereading the paper and rebuttal, I totally agree that task arithmetic comes from weight disentanglement, which could be improved by linear behavior. The authors suggest that sparse fine-tuning leads to linear behavior theoretically whereas I expect some empirical results as the author claimed 'our main focus is the empirical evaluation'.

4. I totally respect the proposed method has a broader focus on task negation, however, the authors have mentioned 'model merging' in sec5 several times, which is a little bit confusing. What's more, it's interesting that authors include Ties-Merging but neglect Ada-Merging which are both during merging methods.

5. Thanks for restating the foundation of the proposed method, we could conclude that [1,2,3] 'focus on over-parametrized pre-trained models already converged to a local optimum for all tasks'. However, after adding 'regularization' how can we confirm that the model still converges to a local optimum for all tasks? There's a clear performance drop on single-task which is a reflection of robustness. Regarding generalization, I referred to different optimizers as the authors claimed 'modify the update rule of any gradient-based optimization algorithm (e.g. SGD)'.

Overall, I hope the authors could address all my concerns above.

We thank the Reviewer for the discussion and the reconsideration of their initial recommendation. In the following we hope to clarify these final concerns.

I acknowledge that the linear behavior results are represented in the appendix; however, this is a significant part of the evidence and should be included in the main text.

We will make sure to dedicate in the final version more space to this aspect in the main text.

In the caption of Table 5, TaLoS has different schemes of model merging, which also indicates that TaLoS fits in the model merging scheme. Plus, it's very interesting to combine TaLoS with adamerging.

Task addition is inherently a model merging scheme, where task vectors are combined using a scaled addition operation. In contrast, TaLoS is entirely agnostic to the choice of model merging scheme (be it Task Addition, TIES-Merging, or AdaMerging). The sole objective of TaLoS is to produce task vectors that when combined guarantee minimal interference, leaving the specifics of their merging or subsequent use beyond its scope.

A good task arithmetic performance comes from both single-task performance and weight disentanglement, […] However, the single-task performance is the upper bound which is a clear bottleneck.

As shown in [1], task arithmetic performance is evaluated across two benchmarks: task addition and task negation (refer to Tables 1 and 2). While single-task performance can serve as an upper bound for task addition, it does not apply to task negation, where the goal is to reduce the zero-shot performance of pre-trained models. Additionally, Figure 9 highlights that other task arithmetic methods also trade off minor reductions in single-task performance, sometimes performing worse than TaLoS (e.g., Linearized FT on ViT-B/32 and Linearized LoRA on T5) for improved task arithmetic performance in both addition and negation. Notably, TaLoS achieves the best performance in these benchmarks (see Tables 1 and 2).

References:

[1] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. "Task arithmetic in the tangent space: Improved editing of pre-trained models." Advances in Neural Information Processing Systems 36 (2023).

We thank the Reviewer for their constructive feedback. We appreciate the positive assessment, particularly regarding the novelty, efficiency and effectiveness of our method. Below, we address the weaknesses identified by the Reviewer and respond to their questions.

• (W1) Sensitivity analysis of low-gradient parameters. We are grateful for the interesting question. Applying a perturbation $\theta_0’ \gets \theta_0 + \delta\theta_0$ to a subset of the pre-trained weights in $\theta_0$ and observing no change in the output $f(x, \theta_0’) \approx f(x,\theta_0)$ intuitively means that those weights have low sensitivity to the task. So, pruning or randomizing them would not affect input-output behavior.
  However, “extreme” randomizations/perturbations would not be suitable to assess the sensitivity of the parameters. If “extreme” randomization refers to very high magnitude perturbations (perhaps, additive), then such perturbations could potentially move the current solution (parametrized by $\theta_0 \in \mathbb{R}^m$) away from the current local optimum, to a distinct region of the loss landscape.
  Indeed, sensitivity analysis generally refers to “robustness to small perturbations”. This concept, alongside how to perform proper sensitivity analysis on the parameters of a neural network has been formalized by a rich literature dedicated to applications of information geometry [1, 2, 3]. Specifically, as shown by [2, 3], to assess the influence of each weight on the output of a network, we can use the Kullback-Leibler (KL) divergence between the output distribution induced by the original network ($p_{\theta_0}$) vs. the one induced by the perturbed network ($p_{\theta_0 + \delta\theta}$). Mathematically, assuming $\delta\theta \rightarrow 0$ (a small perturbation),
  $D_{KL}(p_{\theta_0} \| p_{\theta_0 + \delta\theta}) = \frac{1}{2} \delta\theta^\top F(\theta_0) \delta\theta + \mathcal{O}(\|\delta\theta\|^3)$
  The KL divergence is zero if the perturbation doesn’t affect the output, revealing that the modified weights are not influential for the output. The KL divergence is > 0 otherwise. Here $F(\theta_0) \in \mathbb{R}^{m \times m}$ is the Fisher Information matrix (FIM) [4, 1]. It is a positive semi-definite symmetric matrix defined as,
  $F(\theta\_0) = \mathbb{E}\_x [ \mathbb{E}\_{y \sim p\_{\theta\_0}(y|x)} [\nabla\_\theta \log p\_{\theta\_0}(y|x) \nabla\_\theta \log p\_{\theta\_0}(y|x)^\top] ]$
  It can be used to relate the changes in the parameters to the changes in the outputs. Specifically, it is possible to implement a proper sensitivity analysis of the parameters of a neural network by studying the magnitude of the diagonal elements of the FIM, as they represent the sensitivity of each parameter [2, 3, 5]. Formally, for each parameter $j$, with $j \in 1,...,m$ its corresponding entry on the diagonal of the FIM has value
  $F\_{jj}(\theta\_0) =\mathbb{E}\_x [ \mathbb{E}\_{y \sim p\_{\theta\_0}(y|x)} [\nabla\_{\theta\_j} \log p\_{\theta\_0}(y|x)]^2 ]$
  The higher this value is, the more the $j$-th parameter will be sensitive (i.e. will influence the output of the model). Note that in our work we rank parameter sensitivity using $|\nabla_\theta f(x,\theta_0)|$, which is equivalent to using information from the diagonal elements of the FIM. Modeling $p_{\theta_0}$ as a categorical distribution induced by the outputs of $f$ (which is a proper assumption for multi-class classification tasks), the logarithm function ($y = log(x)$), the squaring function ($y = x^2$) and the absolute value function ($y = |x|$) are monotonically increasing in the open interval $]0,+\infty[$. Hence, the sensitivity ranking is the same as $[\nabla_\theta \log p_{\theta_0}(y|x)]^2$.
  This said, we repeat in Figure 10 (Appendix A.7) the experiment of Figure 1, but by adding noise distributed as $\mathcal{N}(0, 2\sigma I)$ to the bottom-10% of parameters, instead of pruning them. $\sigma$ is the standard deviation of the parameters, previous to perturbation. We report it in the figure together with the average magnitude $\mu$ prior to applying noise. The results align with the analysis reported in Figure 1, highlighting the stability of these parameters across tasks.
  Moreover, we decided to provide further evidence about the overlap of low-sensitivity parameters across tasks in Figure 11 (Appendix A.7). For each parameter, we compute the mean Intersection over Union (mIoU) of masks, between each task pair. Starting from pre-trained parameters $\theta_0$, we predict the mask on task $t$ and then check its intersection over union against the mask predicted on task $t’$ (which acts as a ground truth). A mIoU of 1 signals perfect mask overlap between tasks. The number of parameters selected by each mask is 10%, in line with the experiment of Figure 1.

Answering the questions:

• (Q1) Clarifications about the masking strategy. While TaLoS shares similarities with pruning approaches, we remark that it is not a pruning method. Instead, it is a sparse fine-tuning technique that modifies only a subset of the pre-trained model’s parameters by masking their gradients during optimization (e.g., SGD) (LL228-238). Unlike pruning, TaLoS does not remove parameters from the model.
  The parameters in the subset are those identified as the least important ones for a certain task t with data support $x \in \mathcal{D}\_{t}$, where the concept of “importance” is derived from the output gradient vector $|\nabla\_\theta f(x, \theta\_0)| \in \mathbb{R}^m$ (with the absolute value applied element-wise). For a given data point $x$, the magnitude of the gradient $|\nabla_{\theta_j} f(x, \theta_0)| \in \mathbb{R}$ reflects how strongly the $j$-th parameter influences the output $f(x, \theta_0)$. Larger absolute values indicate greater importance.
  By selecting parameters with consistently small output gradients, we ensure that only those deemed less critical are updated during fine-tuning, while the more influential parameters remain untouched. Unlike pruning methods, TaLoS avoids the risk of inadvertently removing key parameters because it preserves all parameters and selectively fine-tunes the unimportant ones.

• (Q2 + Q3) Clarifications about the pruning experiment of Figure 1. We updated the caption of Figure 1 to be more clear and specific about the chosen pruning criterion, as well as about the sparsity ratio used which is 10%. This value was found as the maximal sparsity that kept the model’s output almost unchanged, when pruning the least sensitive parameters (sensitivity is measured by $\mathbb{E}\_{x}[|\nabla\_\theta f(x, \theta\_0)|]$ with $x$ drawn from the mask calibration dataset).
  Our work builds upon the theoretical and empirical foundations established by [6, 7], which position the pre-training regime as a cornerstone of the task arithmetic framework. In particular, [7] established that task arithmetic is a property acquired during pre-training as this learning process not only leads to semantically disentangled feature representations but also to the disentanglement of the weights that govern the output on those semantic sets. We considered in our work several model sizes under this specific pre-training regime, on both vision and language domains. We empirically observed a parameter-sharing phenomenon for low-sensitivity parameters across tasks: we can identify them, regardless of the data support (see Figure 1 and the discussion in Section LL213-224). This suggests that parameters with low sensitivity to one task tend to exhibit similarly low sensitivity across all other tasks (see also the overlap of such parameters across tasks in Figure 11, Appendix A.7). Notably, model size plays a significant role in this phenomenon: as the model size increases while maintaining a fixed sparsity ratio of 10%, the number of such shared parameters also increases. This is evidenced by lighter regions in Figure 1 for larger models, indicating higher accuracy retention after pruning, and also by the increase in mean Intersection over Union (mIoU) in Figure 11 (Appendix A.7).
  However, we emphasize that this remains empirical evidence. Accurately characterizing the precise conditions and limitations of this parameter-sharing phenomenon for low-sensitivity parameters across tasks, requires further investigation. We leave this as a direction for future work.

Thank you for your detailed response and the additional experiments.

I understand that low-sensitivity parameters have minimal impact and do not interfere with other tasks. However, I am finding it difficult to intuitively understand how fine-tuning such parameters, which have only a small individual influence, can lead to improved performance on specific tasks.

Could you please provide a concise explanation for this? Additionally, could you indicate which figures in the paper best demonstrate this effect?

Addressing these questions would clarify my understanding and make me confident to change my score.

We thank the Reviewer for the observation. We clarify that the parameters identified by TaLoS are not strictly null in influence but rather are comparatively smaller in gradient magnitude to others. As noted in Section 4, we rank parameters based on $|\nabla\_{\theta\_j} f(x, \theta\_0)|$ values and select those with smaller magnitudes, where “smaller” is a relative measure, not an absolute one. Thus, affirming $\nabla\_{\theta\_j} f(x, \theta\_0) \approx 0$ does not imply $\nabla\_{\theta\_j} f(x, \theta\_0) = 0$, reinforcing that even parameters with marginal gradients can collectively contribute to task performance.

This aligns with prior research in parameter-efficient fine-tuning, which has shown that small parameter updates, as well as the involvement of only a limited subset of parameters, can effectively guide learning. For instance, in [1, 2] transfer learning is achieved by fine-tuning only the bias terms of large pre-trained models. In [3] only pre-trained parameters with the smallest absolute magnitude are trainable.

References:

[1] Zaken, Elad Ben, Shauli Ravfogel, and Yoav Goldberg. "Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models." arXiv preprint arXiv:2106.10199 (2021).

[2] Xu, Shichao, et al. "One for many: Transfer learning for building hvac control." Proceedings of the 7th ACM international conference on systems for energy-efficient buildings, cities, and transportation. 2020.

[3] Baohao Liao, Yan Meng, Christof Monz. “Parameter-Efficient Fine-Tuning without Introducing New Latency”, ACL 2023

Dear Reviewer,

as the end of the discussion period is approaching, we kindly ask if our responses have adequately addressed their concerns or if there are any additional questions we can clarify.

We thank the reviewer for their time to review our work and appreciation for the motivations, the simplicity and the effectiveness of our work. We answer to their questions and weaknesses below.

• (W1 + W2) Applicability of the method & Clarification on mask construction. We appreciate the Reviewer’s feedback and recognize that the main paper may benefit from additional details to improve clarity, as some critical information may have been relegated to the Appendix. To this end, we added more precise information at LL245-248 and L260 about TaLoS (including the usage of multiple rounds and a pointer to the mask calibration details and the pseudocode).
  Now, we clarify that TaLoS does not require an additional fine-tuning step and re-state in the following our mask calibration approach.
  First of all, $x$ refers to a single example. We clarified this in the manuscript (LL216, 242, 252, and Algorithm 1). Mask calibration consists of estimating which parameters to sparsely update at fine-tuning time. The calibration runs only once per task before fine-tuning the pre-trained model parameters $\theta_0$.
  As reported at LL239-249 and Algorithm 1, we construct the gradient mask $c \in \mathbb{R}^m$ by setting to 1 the entries in $c$ corresponding to the average output gradient values $s = \mathbb{E}_{x \in \mathcal{D}_t} [| {\nabla}\_\theta f(x,\theta_0) |] \in \mathbb{R}^m$ that are below the threshold $s\_k$. The remaining entries in $c$ are set to 0. The threshold is selected as the $k$-th largest entry in the vector $s$, where $k$ is a user-defined hyperparameter and controls the amount of parameters that will be updated during sparse fine-tuning. $s$ can be obtained efficiently using only a small subset of data of the task by computing per-example gradients. By leveraging vectorized implementations, getting $s$ costs around $M$ forward-backward passes, where $M$ is the number of mini-batches.
  The data used in this process is the validation split of the considered task. As detailed in Appendix A.1 at LL894-899, this choice aligns with the setting adopted by LoTA for mask calibration, ensuring a fair and direct comparison between our calibration method and theirs.
  We remark that the focus is on fine-tuning pre-trained models to produce high-quality task vectors $\tau_t$ that effectively capture the knowledge of every $t$-th downstream task. We can access one task at a time when fine-tuning i.e., we are constrained to individual task training. The data from both the training and validation splits is available in our setting as prescribed by common task arithmetic practices [1, 2].
  As explained in Appendix A.2 (LL1016-1025, and now also at LL245-248 of the main submission), the mask $c$ could be calibrated by performing a single pass on the data split used i.e., in a single round. However, noisy gradients may affect this process, as evidenced in the Pruning-at-Initialization (PaI) literature [3]. A general solution adopted there consists in running multiple rounds and keeping only the bottom-$p$ parameters (where $p \geq k$) at each round, excluding the rest from mask calibration. Thus, we followed that strategy with an exponential decay of the current sparsity $p$. We repeat this process until we reach the user-defined target sparsity $k$, obtaining the final mask $c$. As explained at LL1072-1074, employing around 10 iterations per round, with just 4 rounds (i.e. 40 iterations total) already leads to satisfactory results, as using more rounds/iterations does not yield an increase in the overall performance.
  This means that the fine-tuning happens only once (no need to “redo fine-tuning”).
  Computational cost. Finally, we provide a detailed comparison of the average memory and time costs of TaLoS vs. other fine-tuning methods in the table below. Specifically, we report the timings in seconds (averaged over the 8 vision tasks) and the peak memory usage in Gibibytes of mask calibration and fine-tuning on a CLIP ViT-L/14. These results confirm that paying a small price before training (mask calibration) allows for substantial gains in terms of fine-tuning costs, which also guarantees better task arithmetic performance. By focusing only on the comparison between LoTA and TaLoS we notice that the mask calibration time is approximately the same but the cost in terms of memory is very different. Indeed, the mask calibration of LoTA requires performing a few optimization steps and storing the optimizer states before fine-tuning, while for TaLoS we just need to compute per-example gradients.

Implementation Details for these results: For both LoTA and TaLoS, we used batch size 128 for 40 iterations (in detail, 10 iterations per round for TaLoS, with 4 rounds total) as already specified at LL1072-1074. Moreover, we employed gradient checkpointing (as suggested at L1116) which increases the mask calibration time for both methods by 20% but with a substantial saving in terms of memory. The task arithmetic results are in line with Table 1, with no detrimental effect given by the usage of gradient checkpointing.

• (W3) Mask sparsity level & Single-task fine-tuning results. We thank the Reviewer for pointing out this aspect. In Appendix A.1 (hyperparameter selection paragraph, LL893-895) we explained that the optimal sparsity level for TaLoS was searched in the range [10%, 99%], selecting the best value according to task negation performance as it is a cheap proxy to the degree of weight disentanglement [1]. We are now stating explicitly in the main paper (at LL259-260) that the best results are observed in sparsity ratios between 90% and 99%.
  To provide a more detailed overview of the effect of sparsity on TaLoS we introduced section A.5 in the Appendix. Figure 8 shows the task arithmetic performance vs the single-task accuracy (before addition/negation) while varying the sparsity level. Full (non-linear) fine-tuning corresponds to 0% sparsity. Increasing the sparsity improves the task arithmetic performance, while slightly decreasing the average single-task accuracy, as fewer parameters are updated during fine-tuning. Optimal values for absolute accuracy (in task addition) and target accuracy (in task negation) are observed for a sparsity level of 90% across a variety of models. After 90% sparsity there is a slight drop in both task arithmetic and single-task performance, making such sparsity levels not ideal. Intuitively, if the fine-tuning involves too few weights the resulting entries in the task vector ($\tau_t=\theta_t^\star -\theta_0$) will be mostly zero, reducing the ability to perform task arithmetic effectively.
  We can conclude that, like other parameter-efficient fine-tuning methods, our approach trades some single-task performance for parameter efficiency. But this trade-off allows also for superior task arithmetic capabilities for TaLoS (Tables 1, 2) while maintaining competitive single-task accuracy, especially for larger models where the performance drop becomes negligible (Figure 9).
  Finally, we discuss the single-task performance of TaLoS before task addition as we agree it is important to show these results. We added section A.6 in the Appendix: Figure 9 presents the accuracies obtained by TaLoS (at 90% sparsity) vs. the other fine-tuning strategies.

W8.
We acknowledge the Reviewer’s perspective, but we believe there is a misunderstanding about the aim of our work.
We focus on task arithmetic that by definition encompasses both addition and negation [1, 2]. Up to now, only a single work [1] derived theoretical guarantees and empirical proofs of success in task arithmetic by proposing to fine-tune in the tangent space (via explicit linearization). From that work, [3] improved on efficiency by performing partial linearization of the models via custom low-rank adapters.
Post-hoc methods that perform model merging have so far been narrowly focused, primarily addressing task addition while overlooking task negation (ref to LL112-115).  Although these methods have shown empirical success in task addition, our work is the first to evaluate their performance also in task negation, offering a more comprehensive perspective. We deem it inaccurate to consider post-hoc methods as state-of-the-art approaches in task arithmetic. 

Q2.
The reviewer is right in claiming that, when calling loss.backward(), PyTorch’s automatic differentiation engine goes through the entire computational graph to obtain the gradients for each unfrozen parameter. This happens through the chain rule that applies to activations, not parameters. Even if the parameters of a layer are frozen (requires_grad=False), PyTorch still needs to compute the gradients of the loss with respect to the outputs of that layer. However, it skips entirely the gradient computation with respect to the frozen parameters.

We state it more formally by providing an overview on the chain rule functioning.
Computing derivatives for parameters of the $i$-th layer by applying the chain rule involves the dot product $\frac{\partial L}{\partial w\_i} = \frac{\partial L}{\partial y\_i} \cdot \frac{\partial y\_i}{\partial w\_i}$. Here L is the final loss function, $w\_i$ is a parameter of the layer, and $y\_i$ is the output of the layer. PyTorch just needs to compute and propagate backwards $\frac{\partial L}{\partial y\_i}$, as the derivative with respect to the parameter $w\_i$ is not required and will be skipped, hence, saving computation. In fact, if a previous $i-1$-th layer requires_grad on its weight $w\_{i-1}$, to compute the derivative of the loss with respect to $w\_{i-1}$ it will use the relationship: $\frac{\partial L}{\partial w\_{i-1}} = \frac{\partial L}{\partial y\_i} \cdot \frac{\partial y\_i}{\partial y\_{i-1}} \cdot \frac{\partial y\_{i-1}}{\partial w\_{i-1}}$

Such computation doesn’t involve the term $\frac{\partial L}{\partial w\_i}$. Moreover, both $\frac{\partial y\_i}{\partial y\_{i-1}}$ and $\frac{\partial y\_{i-1}}{\partial w\_{i-1}}$ are “local” computations, involving just the current layer and the following and typically are obtained on the fly using inputs and outputs of the layers, which are cached during forward pass.

As indicated by Figures 4 and 5, for TaLoS most of the layers have zero trainable parameters, which allows to freeze the full layer (in detail, all the MLPs, LayerNorms, Embedding layers and specifically for Vision Transformers the initial Convolutional layer). In the case of attention layers, only the weights producing Queries and Keys are selected (also the biases of Keys in the case of Vision Transformers) which instead enable partial freezing of these layers, i.e., we retain_graph and pass to the optimizer’s parameters argument only $W\_q$, $W\_k$ and $b\_k$ in attention layers. So, while the forward pass still involves the computation for all layers (to calculate the final output), the backward pass skips the gradient computation for frozen parameters, which in the TaLoS case account for more than 80% of the total amount of weights (for sparsity ratios between 90% and 99%). The remaining ~20% of parameters have a mask with at least a 1, so they need to require_grad as gradients need first to be fully computed for the whole parameter and then those gradients get masked before updating the weights (as in Equation 7).

References:

[1] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. "Task arithmetic in the tangent space: Improved editing of pre-trained models." Advances in Neural Information Processing Systems 36 (2023).

[2] Ilharco, Gabriel, et al. "Editing models with task arithmetic." International Conference on Learning Representations (2023).

[3] Tang, Anke, et al. "Parameter efficient multi-task model fusion with partial linearization." International Conference on Learning Representations (2024).

W1+W2.
Thank you for the clarification. We think it is important to consider that knowledge interference reduction in model merging can be performed with two different strategies. Either operating after the single-task training phase using post-hoc techniques, or directly during training. TIES-Merging, TALL Masks, DARE and Breadcrumbs belong to the first family of post-hoc methods and reduce knowledge interference generally following heuristics. Linearized Fine-Tuning, or Parameter-efficient fine-tuning approaches like L-LoRA and LoTA are instances of the second family of methods, as well as TaLoS: they act preventively by modifying the fine-tuning strategy so that interference is less likely to happen during merging. Both types of approaches are valuable and present different advantages/drawbacks: possibly post-hoc methods can be considered more flexible, but fine-tuning approaches when guided by principled parameter selection rules offer more guarantees. Finally, we remark that on the basis of the presented results, TaLoS outperforms all its competitors from any of the described families.

W3.
Regarding task negation, we are not sure why the Reviewer refers to reliability.
In our answer we mentioned that task negation performance is a cheap proxy to the degree of weight disentanglement [1]. Moreover, at L287 in the main paper we pointed the reader to the implementation details in Appending A.1 where we reported our choice regarding this hyperparameter selection strategy (LL880-887).

By measuring the task negation performance we simplify the hyperparameter tuning process. If we have $T$ tasks and want to test $N$ values for a hyperparameter, evaluating task negation requires $T \cdot N$ runs. In case of task addition we should consider all possible combinations among task pairs, requiring $N^T$ runs.

In [1] the authors noted the correlation between normalized accuracy on task addition and target accuracy on task negation. Intuitively, if a task vector is able to achieve better (lower) target task accuracy while retaining high control task accuracy, it indicates that the specialized knowledge captured for that task lies in a subspace with minimal task overlap or redundancy [1, 2, 3]. This enables better preservation of specialized knowledge and avoids destructive interference during task addition. Our findings align with this observation. This is especially evident in Figure 8 where we obtained the same correlation: the lower the target task accuracy in negation, the higher the normalized accuracy in task addition.

[Figure 8] shows, however, that there are cases where the method does not yield any benefits, such as T5. In that case and, in general, providing the normalized performance with bold numbers can be misleading, e.g., Table 1 TaLoS has bold numbers on normalized accuracy 5% higher than traditional Task arithmetic without an actual increase in absolute performance. In this case, the method proposes to not use publicly available checkpoints, but do fine-tuning from scratch after ablating the mask sparsity level, which is not flexible.

Regarding the presentation of results, we are surprised by the Reviewer’s statement. Tables 1 and 2 follow the same format as those in [1], where bold font is used to indicate the best results in each column. When two methods achieve identical results, both corresponding values are appropriately highlighted in bold. We do not see why this should be considered misleading.

With respect to the competitors, TaLoS achieves higher normalized accuracy, as it retains more of its single-task pre-addition performance. This indicates that our method generates task vectors with lower interference compared to all other approaches. Notably, only on T5-Small (Table 1), TaLoS performs on par with Non-linear FT in terms of Absolute Accuracy and outperforms it in Normalized Accuracy. We see no issue of flexibility in this behavior. A user can readily interpret these results and opt for Non-linear FT (e.g., using pre-trained checkpoints) while acknowledging that task interference will result in low normalized accuracy. When scaling to T5-Base and T5-Large, our method demonstrates superior performance in both absolute and normalized metrics.