Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration

Response to NbB2

We sincerely thank you for your critical assessment and constructive suggestions.

Q: Comparison between CAT Merging and LOT Merging.

A: Thanks for highlighting this point. While both CAT Merging and LOT Merging address layer-wise conflicts, they differ fundamentally in their treatment of task vectors:

• CAT Merging uses trimming (projection and masking) to remove “conflict-prone” components. While this suppresses harmful interactions, it may also discard critical task-specific directions, particularly when those directions contain information valuable for other tasks.
• LOT Merging, by contrast, directly minimizes layer-wise feature drift via a convex quadratic formulation. This closed-form solution preserves all informative subspaces and optimally balances contributions from each task. In Section 5.1, we show that in both the ideal (orthogonal features) and worst (collinear features) cases, LOT Merging achieves the optimal consolidation of task vectors—something trimming cannot guarantee.

Empirically, CAT’s trimming sometimes over-prunes useful information, whereas LOT Merging shows more consistent improvements (in Table 10, Section D.7). We will clarify these distinctions in the revised manuscript.

Q: The robustness of LOT Merging under corruption exemplars/test sets.

A: We appreciate the reviewer’s insightful comments regarding the robustness of LOT Merging.

To assess the robustness of LOT Merging under noisy conditions, we adopted the corruption protocol introduced in Ada-Merging [1], applying seven types of corruption to both exemplars and test sets. We evaluated the performance by merging eight ViT-B/32 models on each corrupted dataset and compared LOT Merging to Task Arithmetic. The following results show that LOT Merging maintains consistent improvement over Task Arithmetic under all corruption types.

Q: Results on NLP tasks & the robustness analysis of LOT Merging with different instructions.

A: Thanks for your suggestion. We follow CAT Merging and conduct experiments using RoBERTa as the backbone model on the GLUE benchmark, including eight diverse NLP tasks spanning both classification and regression (e.g., STS-B). For evaluation, we report accuracy for classification tasks and the average of Pearson and Spearman correlations for the regression task. The results are summarized in the table below. Our proposed LOT Merging achieves the highest average performance compared to baseline methods.

Notably, we observe that LOT Merging exhibits superior robustness on tasks with more complex or nuanced instructions, such as:

• rte demands precise logical inference between sentence pairs.
• qqp involves paraphrase detection, relying on intent and semantic understanding rather than surface patterns.
• sst2 requires fine-grained sentiment classification, sensitive to subtle linguistic cues.

In contrast, some baseline methods tend to overfit tasks with simpler instructions, such as stsb, leading to less balanced performance. These results suggest that LOT Merging is more robust across tasks with varying levels of instruction complexity.

Q: The robustness of LOT Merging under real-time data distribution.

We further study the impact of real-time data on model performance, particularly in scenarios where exemplars are available for only a subset of classes, while evaluation is conducted on the complete test set. The results below show that although LOT Merging’s performance declines with fewer exemplar classes, it still consistently outperforms Task Arithmetic.

Q: Effectiveness for larger models.

A: Good question. To assess computational feasibility, we empirically measured the time required to compute the Moore–Penrose inverse of a randomly initialized 12,288×12,288 matrix across several platforms:

These results confirm that the inverse operation is tractable in practice. Even on the Xeon Gold, performing such inversion across 96 layers requires approximately 139.5 minutes. Given the one-time cost and the absence of any retraining, we consider this overhead acceptable at LLM scale. Furthermore, our method only invokes the pseudo-inverse on exemplar-aggregated covariance matrices, which are typically low-rank in practice, further reducing computational burden.

In addition, following [1], we conducted preliminary experiments (due to time constraints) on merging two LLaMA-2-13B-based models: WizardLM-13B, fine-tuned for instruction following (evaluated on AlpacaEval 2.0), and WizardMath-13B, specialized for mathematical reasoning (evaluated on the MATH dataset). As shown in the table below, LOT Merging outperforms Task Arithmetic on both benchmarks, although the margin is relatively modest.

[1] Yu L, Yu B, Yu H, et al. Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement[J]. arXiv preprint arXiv:2408.03092, 2024.

Q: How LOT Merging tackle attention mechanisms or SwiGLU activations.

A: Thank you for your question. Our approach decomposes complex modules into their fundamental parameterized linear operations:

• Attention Mechanism: The Q, K, V projections and the output projection are each treated as distinct linear layers and merged independently. Our experiments with attention-based models such as ViT and BLIP demonstrate the effectiveness of this decomposition.
• SwiGLU Activation: The gate and linear projections in SwiGLU are treated as two separate matrix multiplications, which can be merged independently as well.

This modular treatment allows LOT Merging to naturally extend to attention mechanisms and SwiGLU activations, without requiring any additional modifications.

Q: Where are the exemplar sets selected from?

A: All exemplar sets are randomly sampled from the training set and are kept strictly separate from the test set used for evaluation.

Q: Writing issues (standard deviation and equation error in Section B).

A: Thank you for highlighting these issues. We will thoroughly revise the manuscript, correct the notation, and include the standard deviation in the experiments in the final version.

Response to VNUV

We sincerely thank you for your critical assessment and constructive suggestions.

Q: How does the linear feature drift minimization affect the merging of nonlinear models?

A: Thank you for your thoughtful question. You are correct that the classical closed-form solution is strictly applicable to linear systems. However, our approach leverages the modular structure of modern deep neural networks, which are composed of sequences of linear modules interleaved with nonlinearities. While the overall model is highly nonlinear, we show that it is still possible to bound the output drift of nonlinear modules in terms of the drift in their underlying linear components.

To illustrate this, consider the attention module, a central building block in transformer models. Its computation can be written as:

$Att(Q, K, V) = \mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right) V,$ where $d$ is the feature dimension. We establish the following proposition (with a proof sketch provided below):

---

Proposition 1.

Let $\Delta Q, \Delta K, \Delta V$ denote arbitrary perturbations to the input matrices. Then:

$\|Att(Q, K, V) - Att(Q +\Delta Q, K +\Delta K, V + \Delta V)\|_F$

$\leq \Delta_{linear} + \Delta_{non-linear}$

where $\|\cdot\|_F$ denotes Frobenius norm and

$$\Delta_{linear}=\frac{\|V\|_F \|Q\|_F}{\sqrt{d}} \|\Delta K\|_F+ \frac{\|V\|_F \|K\|_F}{\sqrt{d}} \|\Delta Q\|_F + \|\Delta V\|_F$$ $$\Delta_{non-linear}=\frac{\|V\|_F}{\sqrt{d}} \|\Delta Q\|_F \|\Delta K\|_F$$

---

Proposition 1 shows that the total drift in the attention output—caused by perturbations to $Q$, $K$, and $V$—is bounded by a sum of linear and nonlinear terms. Crucially, when the perturbations $\Delta Q, \Delta K, \Delta V$ are small (which is precisely the regime targeted by LOT merging), the nonlinear term becomes negligible, and the bound is dominated by the linear terms. Furthermore, the coefficients of the linear terms are either constant or normalized by $\sqrt{d}$, ensuring that the overall drift remains controllable.

This bound provides a theoretical justification for our decomposition approach: by controlling the drift in the internal linear layers ($\Delta Q, \Delta K, \Delta V$), we can effectively bound the output drift of the entire attention module—even though the module itself is nonlinear. This insight extends to other nonlinear modules in deep networks, provided we can analyze their dependence on internal linear computations.

---

Proof Sketch of Proposition 1:

Let

$A = \frac{QK^\top}{\sqrt{d}}$

$A' = \frac{(Q + \Delta Q)(K + \Delta K)^\top}{\sqrt{d}}$

$\Delta A = A' - A$

By the triangle inequality,

$\|Att(Q, K, V) - Att(Q + \Delta Q, K + \Delta K, V + \Delta V)\|_F \leq \|\mathrm{Softmax}(A)V - \mathrm{Softmax}(A')V\|_F+\|\mathrm{Softmax}(A')\Delta V\|_F .$

Since each row of $\mathrm{Softmax}(A')$ is a probability vector:

$\|\mathrm{Softmax}(A')\Delta V\|_F \leq \|\Delta V\|_F$.

By the Lipschitz continuity of softmax [1] and sub-multiplicativity of norms:

$\|\mathrm{Softmax}(A)V - \mathrm{Softmax}(A')V\|_F \leq \|\mathrm{Softmax}(A) - \mathrm{Softmax}(A')\|_F \|V\|_F \leq \|\Delta A\|_F \|V\|_F.$

Next,

$\Delta A = \frac{Q \Delta K^\top + \Delta Q K^\top + \Delta Q \Delta K^\top}{\sqrt{d}}$

so

$\|\Delta A\|_F \leq \frac{\|Q\|_F \|\Delta K\|_F+\|K\|_F \|\Delta Q\|_F+\|\Delta Q\|_F \|\Delta K\|_F }{\sqrt{d}}$

Substituting back yields the stated bound.

---

Reference:

[1] Gao, B., & Pavel, L. (2017). On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805.

Q: Would using similar data from other distributions be helpful for LOT merging?

A: Thank you for raising this point.

To investigate this, we merged BLIP models fine-tuned on COCO Caption, OKVQA, and ScienceQA, but replaced each task’s exemplars with samples from semantically related datasets: Flickr30k for COCO Caption, AOKVQA for OKVQA, and IconQA for ScienceQA. As shown below, LOT Merging with surrogate exemplars outperforms Task Arithmetic, though the improvements are task-dependent—substantial for COCO Caption, but more limited for the other two:

We hypothesize that this variability is due to differences in semantic similarity between the surrogate and original exemplars. To systematically test the effect of degraded similarity, we conducted additional experiments using the 8-task vision model merging setup, applying progressively stronger Gaussian blur to the exemplars.

We observe that LOT Merging continues to outperform Task Arithmetic even as the exemplars are heavily degraded, though accuracy decreases as blur increases:

Together, these findings suggest that LOT Merging can indeed benefit from surrogate data, but the effectiveness depends on the semantic similarity to the original task. This insight extends the application of LOT Merging when original data is unavailable.

Response to eNpX

We sincerely thank you for your critical assessment and constructive suggestions.

Q: Does feature drift still have an impact on multimodal tasks?

A: Thank you for raising this important point.

To assess the impact of feature drift in vision-language settings, we measured the Pearson correlation between feature drift and loss change in BLIP models across several multimodal benchmarks after Task Arithmetic merging. The results below demonstrate that feature drift remains a positively correlated factor:

While the correlation strength varies—reflecting inherent complexities such as reasoning demands and modality interactions—our experiments (see Table 3 in the manuscript) consistently show that reducing feature drift leads to performance gains, even in vision-language tasks. This suggests that, despite the greater complexity in multimodal models, minimizing feature drift remains an effective and broadly applicable merging objective.

Q: Can the limited number of exemplars lead to unstable performance?

A: Thank you for the thoughtful comments.

To assess whether limited exemplars affect merging performance, we varied the number of exemplars per task from 1 to 64 and report LOT Merging accuracy on both vision and vision-language tasks. As shown below, increasing the number of exemplars generally improves performance, and most tasks reach near–peak performance by 32 samples. This indicates that the number of exemplars alone cannot fully explain the marginal gain on OKVQA in Table 3.

Regarding the OKVQA results in Table 3, we would like to clarify that LOT Merging only underperforms CAT Merging on this particular dataset. Importantly, LOT Merging improves upon Task Arithmetic across all vision-language tasks, while CAT Merging degrades performance on Textcaps. We therefore attribute CAT’s large gain on OKVQA to task-specific overfitting, rather than to generally superior alignment. A similar observation also applies to our vision-only benchmarks (see Section D.7).

Q: Clarify the individual results of Table 3.

A: Thank you for the suggestion. We will include the individual results in the final version. For reference, the detailed results are as follows:

Response to ZqXr

We sincerely thank you for your critical assessment and constructive suggestions.

Q: Evaluation of the generalization performance.

Thank you for your thoughtful suggestion. To address concerns about generalization, we evaluated our method by merging models trained on six out of eight tasks and then testing on the remaining two unseen tasks.

As shown in the tables below, LOT Merging consistently outperforms both Task Arithmetic and Ties-Merging—not only on seen tasks, but also on unseen (out-of-domain) tasks. These results demonstrate the moderate generalization ability of our approach. However, the performance gap does become narrower on out-of-domain tasks compared to in-domain tasks. This observation motivates us to further explore ways to enhance LOT Merging’s effectiveness on out-of-domain tasks in future work.

Q: Evaluation on large language models (LLMs).

A: Thank you for your suggestion. To address this point, we followed CAT Merging [1] and conducted experiments using RoBERTa as the backbone on the GLUE benchmark, which comprises eight diverse NLP tasks spanning both classification and regression (e.g., stsb). For evaluation, we report accuracy for classification tasks and the average of Pearson and Spearman correlations for the regression task. The results are presented in the table below. LOT Merging achieves the highest average performance compared to the baseline methods, demonstrating its effectiveness for language model merging.

In addition, following [2], we conducted preliminary experiments (due to time constraints) on merging two LLaMA-2-13B-based models: WizardLM-13B, fine-tuned for instruction following (evaluated on AlpacaEval 2.0), and WizardMath-13B, specialized for mathematical reasoning (evaluated on the MATH dataset). As shown in the table below, LOT Merging outperforms Task Arithmetic on both benchmarks, although the margin is relatively modest.

[1] Sun W, Li Q, Geng Y, et al. CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging[C]//ICML2025.

[2] Yu L, Yu B, Yu H, et al. Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement[J]. arXiv preprint arXiv:2408.03092, 2024.

Q: The description of “data-free” may be misleading.

A: Thank you for raising this point. We would like to clarify that we do not describe our method as “data-free” in the paper. Instead, we refer to it as “training-free,” indicating that it does not require any gradient-based optimization. We will make this distinction more explicit in the revised manuscript to avoid any potential confusion.

Q: Exploring the benefit of task-agnostic auxiliary data.

A: Thank you for this interesting suggestion.

To investigate the potential of task-agnostic auxiliary data, we first experimented with using ImageNet samples as exemplars for all tasks. However, when merging eight ViT-B/32 models, we observed a noticeable performance drop compared to Task Arithmetic, as shown in the table below:

Next, we explored the use of domain-similar auxiliary data by applying a Gaussian blur to the original task exemplars. In this setting, LOT Merging consistently outperformed Task Arithmetic across all blur levels, although performance gradually declined as blur strength increased:

These results suggest that fully task-agnostic data can be detrimental to performance, whereas semantically similar data—even when degraded—can still facilitate effective merging. This insight could help extend LOT Merging to more practical or data-constrained scenarios.