CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging

Q4.1: Pay attention to the concurrent work.

A4.1: Thank you for highlighting the concurrent work, "Interfering with Interference: Blind Shuffling and Superposition for Better Multi-Model Compression," which addresses interference during multi-model merging through random layer shuffling and orthogonal transformations.

We appreciate the relevance, as both this concurrent paper and our CAT Merging framework aim to mitigate interference between task vectors. However, there are fundamental differences between their approach and ours:

Motivation and Insight: The concurrent work attributes interference primarily to task vector similarity and proposes randomization-based techniques (layer shuffling and task vector superposition) to increase orthogonality. In contrast, our work explicitly identifies conflict-prone components within task vectors and resolves them through targeted, parameter-specific trimming strategies.

Methodological Differences: Their approach depends on randomized transformations, necessitating task-specific decoding (inverse transformations) at inference time. This requirement limits applicability in scenarios involving mixed-task batches. In contrast, CAT Merging enables seamless multi-task inference without task-specific decoding or per-sample routing, making it more suitable for practical deployments involving shared-task batches.

We will include a discussion of this concurrent study in the related work section of our revised manuscript to position our contributions relative to this relevant work.

Q4.2: What factors lead to the suboptimal performance of CAT Merging?

A4.2: We thank the reviewer for highlighting this important point. While CAT Merging achieves superior average performance—improving accuracy by 2.5% (ViT-B/32) and 2.0% (ViT-L/14) compared to state-of-the-art methods—it is true that it does not always yield the highest accuracy on every individual dataset.

Specifically, we observe that Fisher Merging exhibits better performance on certain datasets (e.g., Cars and SUN397), likely because its weighting mechanism, based on the Fisher information matrix, implicitly prioritizes tasks with weaker performance (larger gradients produce higher Fisher information scores). Conversely, PCB Merging achieves superior performance on datasets like SVHN and MNIST, where masking low-magnitude vector components implicitly favors tasks with stronger fine-tuning outcomes (assuming larger vector magnitudes correlate with greater task specialization).

However, both Fisher and PCB merging tend to perform less consistently across other tasks. In contrast, our CAT Merging framework explicitly targets inter-task knowledge conflicts and aims for a balanced integration across tasks. To illustrate this balance quantitatively, we measured the standard deviation of accuracy drops (defined as the accuracy difference between task-specific models and the merged model) across tasks in the following table. It shows that CAT Merging demonstrates significantly lower variance, reflecting its ability to merge multiple tasks more evenly, without undue preference towards any particular task. We will clarify this analysis and discussion in the revised manuscript.

Q4.3: What is the significance of the removal basis? Is it binary?

A4.3: Thank you for this helpful suggestion. We will clarify this point explicitly in the revised manuscript. Specifically, the removal basis $B$ is a real-valued matrix (not binary). It is optimized to define a task-specific subspace, within which conflict-prone components are identified and suppressed via orthogonal projection. The basis matrix $B$ thus plays a crucial role in effectively mitigating knowledge conflicts during the merging process.

Q1.1: Is the Lipschitz continuity assumption becoming less reliable in Transformer architectures?

A1.1: We thank the reviewer for this insightful observation. Indeed, the multiplicative interactions in Transformer architectures complicate the Lipschitz continuity assumption. However, given that both the network parameters and the input data are practically bounded, we can still derive a sufficiently large Lipschitz constant, under which Theorem 5.1 remains valid—albeit with a looser upper bound. Importantly, we would like to emphasize that the primary role of Theorem 5.1 is to provide theoretical insight and motivation for our layer-by-layer trimming strategy, rather than deriving tight practical bounds. Please see response A2.6 for additional details.

Q1.2: Is the trimming method proposed to solve the magnitude issue?

A1.2: Not exactly. The proposed trimming method specifically addresses knowledge conflicts during model merging rather than magnitude issues alone. Magnitude-based techniques (e.g., Ties-Merging) attempt to resolve conflicts by masking low-magnitude components. However, as illustrated in Fig. 1, simply masking low-magnitude components does not fully eliminate conflicts, since high-magnitude components can also cause significant interference. In contrast, our CAT Merging explicitly identifies and trims components based on their actual contribution to knowledge conflicts, rather than simply their magnitude. This targeted strategy more effectively mitigates conflicts, leading to improved merging performance.

Q1.3: Does CAT Merging have a high computational complexity?

A1.3: The computational overhead of CAT Merging is reasonable and practically efficient. Specifically, CAT Merging involves two main steps:

1. Feature Extraction: This step is lightweight and efficient, requiring only a small number (2–3 per task) of unlabeled samples.
2. Eigendecomposition: While eigendecomposition has theoretically higher computational complexity, in practice, we efficiently mitigate this through GPU parallelization. Moreover, CAT Merging only requires the eigenvectors corresponding to the top-c (2-4 in our work) eigenvalues, enabling further acceleration through specialized methods (e.g., torch.lobpcg). Empirical results (provided in the table below, measured on a single RTX3090 GPU in seconds) demonstrate that CAT Merging performs much faster than training-based counterparts (e.g., TA w/ Surgery, AdaMerging).

Q1.4: Could activation functions simplify the trimming matrices?

A1.4: We agree that a deeper analysis of activation functions would be beneficial. While CAT Merging does not explicitly model activation functions, it implicitly captures their effects through the layer-wise trimming strategy. For instance, activation functions such as ReLU can cause certain dimensions to consistently remain inactive. Referring to Section 5.2 of our paper, the trimming mask for scaling parameters is computed as: $\sum_{i\neq k} \left( \sum_{x^l_k} ( x^l_k\circ T^l_i )^2 - \lambda \sum_{x^l_i} ( x^l_i\circ T^l_i)^2 \right)$ If the $d$-th dimension remains consistently inactive (i.e., $x_k^l[d] \equiv x_i^l[d] \equiv 0$), the corresponding element in the trimming mask naturally becomes zero. Thus, activation functions indirectly simplify the trimming process, even though they are not explicitly modeled in CAT Merging. We will clarify this point explicitly in our revised manuscript.

Q1.5: Why should only a very low-dimensional subspace (2-4 dimensions) be trimmed as suggested in Figure 4(b)?

A1.5: In Figure 4(b), trimming only a low-dimensional subspace (2–4 dimensions) is sufficient because knowledge conflicts are predominantly concentrated within a few principal dimensions. Specifically, as shown in eign.png, the first few eigenvectors represent the most significant directions, accounting for an average of 78.56% (ViT-B/32) and 87.28% (ViT-L/14) of the total eigenvalues. Thus, while conflicts are indeed severe, their severity primarily manifests along these critical dimensions.

Moreover, trimming a higher-dimensional subspace risks unnecessarily degrading the original task performance, as it could remove important task-specific information. Therefore, selecting this low-dimensional subspace effectively balances conflict mitigation and preservation of model performance.

Q1.6: λ in Figure 4(a) should be tuned in the range of (0, 1).

A1.6: As suggested, we conducted additional experiments with λ values in the (0, 1) range. The results, provided in sensitive-lambda.png, indicate that model performance consistently improves as λ increases from 0 toward 1, peaking around 5.

Q2.1: Results on LLM.

A2.1: Thanks for your suggestion. We conducted additional experiments using RoBERTa as the backbone model on the GLUE benchmark. As summarized in A3.3 below, CAT Merging consistently achieves superior average performance compared to existing state-of-the-art merging methods, demonstrating its effective generalization and robustness in language model merging scenarios.

Q2.2: Should all baselines from Tables 2 and 3 be included in Table 4?

A2.2: Thank you for this suggestion. Table 4 is intended as an ablation study specifically designed to analyze individual components within our proposed method; therefore, including all baselines from Tables 2 and 3 may not align well with its purpose. Could you please clarify whether you instead suggest adding all baselines to Table 3? If so, we will gladly incorporate the additional baselines to ensure a comprehensive and thorough evaluation.

Q2.3: How do the proofs in Appendices A.3, A.4, A.5 “imply top c eigenvalues or components”, and how is c optimized?

A2.3: Thank you for pointing out this issue. Specifically, the parameter c is a practical, data-dependent hyperparameter chosen to balance performance stability and exemplar efficiency. Given a fixed c, the optimality of selecting the top c eigenvectors can be rigorously justified. For instance, Eq. (21) in A.3 leads to the following optimization form: $Tr(B^\top GB)=\sum_d \lambda_d ||B^\top v_d||^2$, This is a well-known optimization problem whose solution is directly obtained via the Courant–Fischer theorem, establishing that the optimal choice of $B$ consists precisely of the eigenvectors corresponding to the top c eigenvalues of $G$. We will provide a clearer, step-by-step derivation of this result in the revised appendix.

Q2.4: Modify the motivation to highlight the important task directions instead of magnitude issues.

A2.4: Thank you for this insightful suggestion. We will clarify our motivation to reflect that CAT Merging explicitly identifies and trims conflict-prone components based on their directional contributions to knowledge conflicts rather than relying solely on vector magnitudes.

Q2.5: Should not assume α=1.

A2.5: Thank you for raising this point. The assumption of α=1 was introduced solely to simplify the theoretical analysis, as the scaling factor α does not affect the resulting conclusion. For example, in Eqs. (9)&(10), if we explicitly include α, the optimal vector B corresponds to the eigenvector of the matrix: $\sum_{i\neq k} {(\alpha T_i)}^\top( {X_k}^\top X_k - \lambda {X_i}^\top X_i) {(\alpha T_i)}=\alpha^2\sum_{i\neq k} {T_i}^\top( {X_k}^\top X_k - \lambda {X_i}^\top X_i) {T_i}$. Since scaling a matrix by a nonzero constant α² does not alter its eigenvectors, our conclusions remain valid. Nevertheless, we agree with the reviewer that, in practical model merging, α is a critical hyperparameter that must be tuned carefully. We will provide necessary clarifications in the revised manuscript.

Q2.6: Isn't the upper bound vacuous in Theorem 5.1?

A2.6: We acknowledge that the bound presented in Theorem 5.1 is relatively loose. However, this bound is primarily intended to provide conceptual insight into the motivation underlying our layer-wise trimming strategy. Specifically, Theorem 5.1 establishes the inequality:

$| L(W) - L( W + \Delta W) | < \beta \sum_{l=1}^L \Bigl(\prod_{m=l+1}^L \gamma_m\Bigr) \|\Delta f^l(W^l) - f^l(W^l + \Delta W^l) \|$.

This inequality highlights that reducing layer-specific conflicts (the right-hand side) directly contributes to controlling the difference in model performance (the left-hand side). Thus, even though the bound itself is not tight, it provides theoretical justification for our subsequent conflict-aware trimming strategies at each layer.

Q2.7: Comparison of runtimes.

A2.7: Please see A 1.3 in our response to C9mN.

Q2.8: Figure 2 needs improvement.

A2.8: Thanks. We update Figure 2 in sensitivity-experiment.png, which now more clearly demonstrates that our method remains stable in a rational range, with respect to both the number of exemplars and the value of α.

Q2.9: How are the exemplars selected in Figure 2?

A2.9: The exemplars are selected randomly, and we report the average performance of three random runs.

Q2.10: Do “shift” parameters mean biases?

A2.10: Yes. They include both the bias parameters of linear layers and the shift parameters in normalization layers. Since they share the same trimming form, we treat both of them as “shift” parameters for brevity.

Q2.11: Are attention layers treated as linear layers?

A2.11: We decompose it into several linear layers and treat them separately (e.g., three linear layers for Q/K/V calculations).

Q2.12: Writing issues (verbose formulations and citations).

A2.12: We will revise accordingly in the final version.

Q3.1: Writing issues (W1-4).

A3.1: Thanks for the suggestions. We will revise them and thoroughly double-check the manuscript to avoid similar issues.

Q3.2: Comparisons on inference speed and computational overhead (W5).

A3.2:

The inference speed remains consistent with that of the individual models, as CAT Merging produces a unified model to conduct inference without adding extra layers.

The computational overhead of CAT Merging is reasonable and practically efficient. Specifically, CAT Merging involves two main steps:

1. Feature Extraction: This step is lightweight and efficient, requiring only a small number (2–3 per task) of unlabeled samples.
2. Eigendecomposition: While eigendecomposition has theoretically higher computational complexity, in practice, we efficiently mitigate this through GPU parallelization. Moreover, CAT Merging only requires the eigenvectors corresponding to the top-c (2-4 in our work) eigenvalues, enabling further acceleration through specialized methods (e.g., torch.lobpcg). Empirical results (provided in the table below, measured on a single RTX3090 GPU in seconds) demonstrate that CAT Merging significantly outperforms training-based counterparts (e.g., TA w/ Surgery, AdaMerging) in terms of computational efficiency.

Q3.3: Comparisons of LLMs (W6 and W8).

A3.3: Thanks for your suggestion.

To further validate CAT Merging on language tasks, we conducted additional experiments using RoBERTa as the backbone model on the GLUE benchmark, which comprises eight diverse NLP tasks, including classification and regression (STS-B). We report accuracy for classification tasks and the mean of Pearson and Spearman correlations for the regression task. As summarized in the table below, CAT Merging consistently achieves superior average performance compared to existing state-of-the-art merging methods, demonstrating its effective generalization and robustness in language model merging scenarios.

Q3.4: The theoretical assumptions do not capture the complexities of real-world scenarios (W7).

A3.4: We agree with the reviewers that theoretical assumptions, such as Lipschitz continuity, may not fully capture the complexities encountered in real-world scenarios. Nevertheless, we would like to emphasize that the primary role of Theorem 5.1 is to provide theoretical insight and motivation for our layer-by-layer trimming strategy, rather than deriving tight practical bounds. For further details, please see our response A2.6. In the final version of our manuscript, we will clarify this point further and include an explicit discussion on the potential limitations of our method.

Q3.5: Are the projection and masking the best strategies (W9)?

A3.5: Good question. Our proposed strategies—projection for linear weights and masking for normalization parameters—are empirically motivated heuristics chosen for their computational efficiency and practical effectiveness in mitigating parameter conflicts. While these methods achieve strong empirical results, we acknowledge they are not theoretically guaranteed to be optimal. In future work, we aim to explore principled approaches, such as weighted averaging or Mixture-of-Experts (MoE), to further refine and theoretically ground our conflict mitigation techniques.

Q3.6: Comparing with training-based methods (W10).

A3.6: Thank you for this suggestion. The following table shows that our CAT Merging achieves comparable or superior performance relative to two representative training-based techniques, demonstrating its effectiveness without incurring additional computational costs.