No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces

We are pleased that the Reviewer appreciates the soundness of our introduced metrics, the simplicity and effectiveness of the proposed approaches, and clear writing. We thank the Reviewer for the comments and we respond below to specific points.

[Reference 1 (R1)]: Section 4.2, which discusses retaining components from the common subspace and the orthogonal projection in Equation 10, bears resemblance to the shared subspace optimization concept in DOGE [2]. It is recommended to discuss this.

Thank you for pointing us to this recent reference. We were not aware of DOGE paper at the time of preparing a submission as the preprint appeared two weeks before the deadline. Here we compare this approach with ours:

• Definition of common/shared subspace: We define the common subspace as top-k components from sum of individual task matrices. DOGE defines shared as concatenation of top-k components from each task matrix followed by SVD, which resembles the TSV method.
• Orthogonal projection: Both Iso-CTS and DOGE use an idea of orthogonal projection. DOGE uses it on the gradient of $\Delta$ to restrict the optimization process from changing the shared space. Iso-CTS uses the orthogonal projection on the level of weight matrices to determine the task-specific subspace that is orthogonal to the common subspace.

Iso-CTS and DOGE both use ideas of common/shared subspace and orthogonal projection in different ways. Moreover, the results of our approaches are better than DOGE. We believe that this discussion is very significant and we will add a detailed version of it to the revised manuscript.

[Weakness 1 (W1)]: Lack of further analysis, such as why SVD can be used for model merging due to the redundancy of parameters caused by fine-tuning. It now seems more like an experimental discovery, and these further analyses would elevate the article to a higher level.

It is known that fine-tuning of large pre-trained models results in low-rank parameter update. This observation enables efficient fine-tuning of models using inherently low-rank adaptation techniques such as LoRA. Consequently, the recent TSV paper shows how low-rank approximation of the parameter update matrices, obtained using SVD, can be used to facilitate model merging.

In our paper, we propose to extend the scope of SVD-based analysis for the purpose of model merging. Most importantly, by introducing the SAR metric we show that SVD can help in understanding the overlap between task-specific and merged matrices. Moreover, we show that by modifying the spectrum of singular values of merged matrix we can increase the alignment between task and merged matrix (see the Response to Reviewer ff9S (section Q2) for detailed discussion).

[Suggestion 1 (S1)]: Supplement experiments on NLP tasks to verify the generalizability of the method and make the article more complete.

We present NLP results in response to Reviewer mnUL (section C3). Iso-C and Iso-CTS outperform other baselines across two presented settings.

[Question 1 (Q1)]: Why can subspace alignment eliminate conflicts and improve performance?

Consider the Subspace Alignment Ratio between a task matrix and merged task matrix. SAR quantifies an overlap between the subspaces spanned by dominant singular vectors of these matrices. If the SAR is low, the overlap between the subspaces of these matrices is low and the corresponding singular vectors are close to orthogonal. Therefore, the merged matrix cannot reliably represent the dominant components of task matrix if SAR between them is low. Therefore, low SAR leads to low performance on the corresponding task. Conversely, high SAR indicates high subspace overlap, indicating that the merged matrix can reliably represent the important components of the task matrix and results in high performance.

[Q2]: How is the ratio of singular values controlled between the common and task-specific subspaces?

The ratio of singular values between the common and task-specific subspaces is controlled by the hyperparameter $k$ that is fixed for all the experiments. $k$ is chosen such that $k/r$ for a single layer is equal to 0.8. The final paragraph of Section 5.3 from the paper contains an analysis of impact of $k$ on performance.

We are pleased that the Reviewer appreciates the novelty of the proposed method, the significance of our contribution, and the clear writing. We thank the Reviewer for their constructive feedback, and below we respond to specific points raised.

[Question 1 (Q1)]: In Fig.2, the author tries to show...

Providing Fig. 2(a) along with only the Normalized Accuracy Improvement (NAI) for each task would show that all task vectors $\Delta_t$ exhibit similar cosine similarities close to zero, yet the NAIs vary significantly (e.g. the DTD task vector is orthogonal to all others but has the lowest NAI). We agree that this alone suggests that mere orthogonality among task vectors does not explain differences in performance.

Fig. 2(b) however takes this analysis one step further by examining whether the cosine similarity between each $\Delta_t$ and the task arithmetic vector $\Delta_{TA}$ correlates with NAI. Intuitively, one might expect that a higher similarity between $\Delta_t$ and $\Delta_{TA}$ would result in a higher NAI, but no clear correlation is found. This reinforces the observation that cosine similarity is not a good predictor of performance improvement. We conducted this analysis because we believe that a key factor in understanding the effectiveness of task arithmetic model merging is to directly compare each task matrix with the merged model matrix. Since cosine similarity between task vectors and task arithmetic vector alone does not explain performance gains, we propose analyzing the Subspace Alignment Ratio (SAR) between each individual and merged task matrices, which indeed shows a positive correlation with performance improvement (Fig. 3). In the revised manuscript we will include elements of this discussion to more clearly link the results in Figs. 2 and 3.

[Q2]: I may be missing or misunderstanding something, but I have a question regarding the motivation of the proposed method...

The subspace alignment ratio $SAR_\text{avg}(\Delta_t, \Delta_{TA})$ quantifies how well a task matrix $\Delta_t$ is represented by the subspace of the task arithmetic matrix $\Delta_{TA}$. The subspace dimension, $k_M$ as defined in Eq. (6), is determined by the number of singular vectors required to minimize the reconstruction error in terms of Frobenius Norm. Because the singular value spectrum of $\Delta_{TA}$ is skewed, only a few singular values are large, leading to a low $k_M$ (see Fig. 4(a) - $\beta=0.0$ (TA), marked by a vertical red dashed line). Relying on these few singular vectors to represent each task matrix produces a highly variable $SAR_{avg}$ across tasks (Figure 4(b) - $\beta=0.0$), indicating that some tasks are not well captured by this limited subspace.

The motivation for "flattening" the singular values via Iso-C in Sec. 4.1 is to address this issue. By scaling the singular values, the influence of less dominant singular values increases while that of the dominant ones decreases. This adjustment raises the effective subspace dimensionality $k_M$ (as shown by the vertical dashed lines for $\beta > 0$ in Figure 4a), resulting in a subspace that better represents all task matrices. Consequently, this leads to a higher $SAR_{avg}$ (Fig. 4(b)) and Normalized Accuracy Improvement (Fig. 4(c)). Thus, the skewness in Fig 1(a) explains the variability in $SAR_{avg}$ (and hence merging performance), and the singular value flattening is introduced as a solution to this limitation.

[Q3]: Some important baselines are missing in the experiments, such as Fisher merging, RegMean, and a recent SOTA PCB-merging...

Thank you for pointing out PCB method which is a recent and relevant baseline. Originally, we omitted Fisher Merging and RegMean for brevity as they are outperformed by many recent methods. We additionally include the CART baseline requested by Reviewer BR4L. The PCB paper reports the average absolute accuracy for merging 8 tasks across 2 model sizes and we compare these results in the Table below:

Iso-C and Iso-CTS outperform all of the added baselines.

[Q4]: What are the values of $k$ in Tab.1?

We use $\frac{k}{r} = 0.8$ as a default for all the experiments (see L408-410, right column), where $r$ is a number of singular values for a given layer. Therefore, $k$ can vary across layers according to the $r$ for each particular layer.

[Q6]: ...I am curious about the performance of Iso-CTS w/o line 12...

We present the comparison of the performance of Iso-CTS and Iso-CTS w/o line 12: https://imgur.com/a/39CrGKJ. We observe that isotropic scaling is indeed a crucial component of Iso-CTS. However, the design in Alg. 2 also plays an important role, especially when the number of merged models increases, leading to up to 2.8% improvement on 20 tasks (see Table 1).

We are glad that the Reviewer appreciates the soundness of the experimental protocol and analyses, the effectiveness of the proposed approach, and clear writing. We thank the Reviewer for their comments and we respond below.

[Weakness 1 (W1)]: Lacks theoretical justification...

We provide a detailed explanation why making merged matrix isotropic increases subspace alignment in the response to Reviewer ff9S (section Q2).

[W2]: SVD baseline is insufficient.

Thank you for pointing out CART which is a recent and very relevant baseline. We added it (alongside baselines suggested by Reviewer ff9S, section Q3) to our main Table for 8 tasks (as these are the only results reported in the CART paper):

Iso-C and Iso-CTS outperform all these methods.

[W3]: Experiments on a single model family.

In this rebuttal we add NLP experiments highlighting the effectiveness of Iso methods on the T5 -- an encoder-decoder language transformer (see the response to Reviewer mnUL, section C3).

[W4]: ...Evaluating the approach on PEFT (e.g. LoRA)...

Thank you for the suggestion - it helps to emphasize the generalizability of our approach. We follow the evaluation protocol of KnOTS[1], a recent SOTA (ICLR 2025) method for merging LoRA fine-tuned models, tested on 8 vision tasks using ViT-B/32 and ViT-L/14. For comparison, we merge the task-specific LoRA weights - provided by the authors - to the pre-trained models, and then we apply Iso-C and Iso-CTS. Below, we present the average normalized accuracy:

Iso-CTS achieves SOTA results in LoRA merging setting. Note that our method is a general purpose merging technique while KnOTS is specifically designed for the LoRA merging. This highlights the versatility of Iso methods.

[1] Stoica et al. Model merging with SVD to tie the Knots, ICLR 2025.

[W5]: The methods have not been tested in the NLP domain.

See our response to Reviewer mnUL (section C3) for NLP results.

[Comment 1 (C1)]: ...can authors provide a complexity analysis?

Let $\Delta_t \in \mathbb{R}^{m\times n}$, with $m\geq n$ and let $T$ and $L$ be the number of tasks and network layers, respectively. For simplicity, assume that each layer has a single matrix, whose dimensions are $m$ and $n$. In the analysis below, the lines refer to Algorithm 2 in the main paper.

• Iso-CTS: One SVD on $\Delta_{TA}$ (lines 2-3) whose complexity is $O(mn^2)$ and this is applied to each layer, so the complexity is $O(Lmn^2)$; one SVD on each $\Delta_{t}, t= 1..T$, for each layer (line 5) so the complexity is $O(LTmn^2)$; finally line 11 requires two SVDs on matrices $U \in \mathbb{R}^{m \times r}$ and $V \in \mathbb{R}^{n \times r}$. Since $m \geq n$, then $r=n$, the complexity is $O(2Lmn^2)$. The total complexity is: $O(IsoCTS)=O(Lmn^2+LTmn^2 + 2Lmn^2)=O(LTmn^2).$
• Iso-C: One SVD on $\Delta_{TA}$ with complexity: $O(IsoC)=O(Lmn^2).$
• TSV (our nearest competitor): $T$ SVDs per layer on each task matrix (line 1 Alg. 1, TSV paper), and two additional SVDs per layer(line 10-11 Alg.1 TSV paper) and thus: $O(TSV)=O(LTmn^2+2Lmn^2)=O(LTmn^2).$

While Iso-CTS and TSV share the same asymptotic complexity, Iso-CTS incurs slightly more overhead due to the SVD on $\Delta_{TA}$ (lines 2-3). Both methods can be further optimized by computing Truncated SVDs for Iso-CTS (line 7) and TSV (line 1 Alg. 1, TSV paper), since only a few components are retained. This reduces the complexity for both approaches. Iso-C is the most computational efficient algorithm.

[C2]: Why Table 1 only reports “average absolute accuracy” and “average normalized accuracy” but not NAI?

We report these two metrics to stay consistent with previous literature (Consensus TA, TSV-M). In the revised manuscript we will add a Table reporting NAI.

[C3]: The authors should provide more motivation...

Thank you for the suggestion. In the revised manuscript we will put more emphasis on "no task left behind" achieved by Iso methods highlighting higher performance improvements for tasks underrepresented in TA.

[Question 1 (Q1)]: The claim in Section 3.2...

Yes, each pair of task vectors has a near-zero cosine similarity (Fig. 2a). However, our analysis goes a step further by comparing the cosine similarity between individual task vectors and the task arithmetic vector, demonstrating that this measure alone does not correlate with normalized accuracy improvement (Fig. 2b). Since cosine similarity alone does not explain performance gains, we introduce SAR. Unlike cosine similarity, SAR allows for meaningful differentiation among task matrices by highlighting shared subspaces (Fig. 3b). Additionally, SAR positively correlates with NAI (Fig. 3a).

We are pleased that the Reviewer acknowledges the novelty of the proposed method, the contribution towards understanding model merging and clear writing. We thank the Reviewer for the constructive feedback, and below we respond to specific points raised.

[Minor weakness 1 (MW1)]: Sec. 3, along with Fig. 2 and 3, lacks details of the experimental setting and the models considered.

We used ViT-B/16 and 8 tasks (see Sec. 5.1 for details). We will clarify this in the revised manuscript.

[MW2]: $\rho$ (L195, right) is undefined in the main text.

$\rho$ is a Pearson correlation coefficient – defined in the caption of Fig. 3. We will unify the notation to $\rho_{\text{TA}}$ and add the definition to the main text.

[MW3]: L240-246 repeatedly mention more/less correlated tasks. I think this terminology is vague and should be clarified in terms of alignment.

We will clarify this part in the revised manuscript to avoid vagueness:

However, significant variability in the average alignment ratio across the dataset leads to a lower accuracy improvement for less aligned tasks compared to the tasks belonging to groups of high alignment. This variability stems from the skewness of the task arithmetic spectrum (Fig. 1), which is concentrated in the first few singular values (which we call top or dominant), favoring the tasks from the highly aligned groups.

[MW4]: The claim in L256-258 (right) would benefit from explicit justification.

We can formalize the SVD problem for the first left principal singular vector as the variance maximization problem:

$u_1=\arg\max_{||u|\vert=1}||\Delta_{TA}^Tu||^2=u^T\left(\sum_{t=1}^T\Delta_t\Delta_t^T\right)u+u^T(\sum_{{t,s=1,t\neq s}}^T\Delta_t\Delta_s^T)u$

If a particular task $\Delta_j$ has dominant directions with significantly lower intensity compared to the other tasks (i.e. lower Frobenius Norm), then its individual contributions $\Delta_j \Delta_j^T$ to the total variance becomes smaller. Similarly, cross terms involving $\Delta_j$ will also be comparatively small. Therefore, task $j$ explicitly contributes less to the maximized variance captured by the first principal singular direction.

Moreover, if the directions of $\Delta_j$ are orthogonal or nearly orthogonal to $u_1$, (i.e. $u_1^T\Delta_j=0$), task $j$ contributes minimally or not at all along this principal direction. Similar considerations apply to subsequent singular vectors $u_2, \ldots u_k$, defining the common subspace. Finally, as the number of tasks $T$ increases and tasks become more diverse, it becomes increasingly likely that tasks with distinct but smaller-magnitude directions will be underrepresented or absent in the dominant singular directions identified by the task arithmetic decomposition. This is empirically supported by our results. Iso-CTS provides the most improvement when the number of tasks increases.

[Comment 1 (C1)]: Have you analyzed SAR at different depths of the models?

We analyze SAR at different depths. For the ViT-B/16 model, we calculate SAR between fine-tuned and merged weight matrices and an average of all the matrices from a given layer. We present the results here: https://imgur.com/a/tLnEoAi. We observe that the alignment is higher for Iso across all layers of the vision transformer. One may expect early layers to be more aligned but we find that for both approaches the alignment is similar across the layers.

[C2]: I assume Fig. 3(a) is obtained for a ViT base. How would it look for the larger model?

Yes, we obtain Fig. 3(a) for ViT-B/16. See the Figure for ViT-L/14 here: https://imgur.com/a/3V6xv7T. It closely resembles Fig. 3(a) for ViT-B/16 from the paper.

[C3]: I think the paper would be much stronger presenting some results for NLP...

We present NLP results following the experimental setup from [1]. We use T5-Large-LM-Adapt base model fine-tuned on tasks from T0 mixture. We consider subsets of 8 and 7 NLP tasks adhering to the setup from Table 1 from [1] and compute an average accuracy of Iso-C and Iso-CTS in these settings:

Both Iso-C and Iso-CTS significantly outperform the competing approaches, which highlights the versatility of our proposed methods. We observe that Iso-CTS achieves slightly worse results than Iso-C. This is consistent with our vision results in which both approaches performed very similarly when merging 8 models. We argue that the common space captures all the directions necessary to reliably represent these 7 and 8 NLP tasks, while task-specific subspaces may became more effective when merging more models.

[1] Tam et al. Merging by Matching Models in Task Parameter Subspaces, TMLR 2024