CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging

Thanks for your valuable feedback and constructive suggestions! We've provided additional experimental tables in Anonymous Link. Below, we address your main concerns:

Q1: Comparison with recent baselines (Localize-and-Stitch [1], TALL-masks [2]).

We experimentally compared CABS with TALL-masks and Dataless Localize-and-Stitch, with our results clearly demonstrating the superior performance of CABS.

For TALL-masks, please refer to our detailed response to Reviewer QfMa (Q5) for a comprehensive comparison.

For Dataless Localize-and-Stitch, experimental results are presented in Tables 4, 5, and 6 of the anonymous link. When merging 2, 4, or 6 models, its performance is often close to the base model, indicating limited effectiveness. We found it essentially equivalent to simplified TIES-Merging (fixed λ=1, no 'elect').

This setting works reasonably well in the original setup, where 12 models are merged using 5–10% sparsity, so that λ = 1 happens to act as a good approximation (e.g., 5-10% × 12 ≈ 0.6-1.2). However, when merging a smaller or larger number of models, such a fixed λ becomes suboptimal.

If desired, we are happy to include a λ-tuned version of Dataless L-and-S for fairer comparison.

Q2:Effectiveness of "elect" step in TIES.

We appreciate the valuable ideas introduced by TIES-Merging, particularly its insight and solution regarding parameter redundancy and conflict. Inspired by TIES-Merging, CABS offers a more robust alternative.

Specifically, the “elect” step in TIES is designed to resolve conflicting signs between task vectors and can indeed be helpful in certain cases. For example, as shown in Table 17 of our appendix, TIES-Merging is the best-performing baseline after CABS. However, its effectiveness is inconsistent. The TIES paper itself (Appendix B.1, Table 7) shows that re-estimating the sign vector via few-shot training improves performance, indicating that its elect strategy may be suboptimal.

Our experiments show similar trends. As seen in Table 1, the performance gap between Task Arithmetic+Magnitude and TIES-Merging is often small despite the difference being the elect step, and in some cases, TIES-Merging even underperforms.

One possible explanation is the significant imbalance in average magnitudes among task vectors—up to a 10× difference as illustrated in Table 8 of the anonymous link. In such cases, the sign vector may be dominated by the task with the largest magnitude, resulting in biased merged results.

Overall, while TIES-Merging laid important groundwork in addressing parameter conflicts, CABS advances this direction by providing a more robust and effective solution.

Q3: Are parameter interference the right problem？

As noted in our response to Reviewer QfMa (Q5), CABS and TALL-masks operate under fundamentally different paradigms.

While TALL-masks separates tasks during inference and does not explicitly address parameter interference, it incorporates TIES-Merging as the first stage in its pipeline. This design choice implicitly acknowledges parameter interference as a key factor.

In such settings—where task identity is unavailable and generalization is required—parameter conflict becomes an inevitable challenge for training-free model merging methods.

Q4: motivation for choosing n:m pruning.

We choose n:m pruning for two reasons: (1) Prior methods, like row-wise (topk), already outperform layer-wise pruning, making it natural to explore more structured forms, like block-wise (n:m), to further improve merging performance. (2) n:m pruning operates at the finest granularity (weight level), is simple to apply, and requires no changes to model architecture.

Other pruning techniques are less suitable for training-free merging. Coarse-grained methods (e.g., head pruning) cannot remove redundant weights or resolve conflicts within components. Sparse fine-tuning and similar methods require training, limiting their practicality.

Q5: Potential scalability concerns for CA.

Please refer to our detailed response to Reviewer zNot (Q4).

Q6: Analysis on the impact of pruning order.

Please refer to our detailed response to Reviewer QfMa (Q6).

Building on our new findings on task vector magnitudes (Table 8 in the anonymous link), we suggest that placing task vectors with smaller magnitudes earlier in the pruning sequence—or assigning them larger λ—may help improve balance. This could be a promising future direction, as exhaustive search is infeasible in LLM merging.

Q7: necessary of Validation data usage for tuning λ.

Like TA, TIES, and DARE, we use validation data for tuning λ. The specific values are provided in Appendix B.11.

Q8: Reporting statistical significance with confidence intervals.

Please refer to our detailed response to Reviewer QfMa (Q4).

We hope these responses address your concerns. Please feel free to let us know if anything remains unclear.

Thanks for your valuable feedback and constructive suggestions! We've provided additional experimental tables in Anonymous Link. Below, we address your main concerns:

Q1: Whether the observed performance gap is due to lack of rescaling in magnitude-based pruning.

We agree that rescaling is a key factor contributing to DARE’s effectiveness and could enhance the performance of magnitude-based pruning methods. In our experiments, all magnitude-based pruning methods were equipped with rescaling. The λ values used for merging (e.g., λ=1.88 for 4-model TIES-Merging) are detailed in Appendix B.11, confirming that proper rescaling was employed. Additionally, we also conducted rescaling experiments for magnitude-based pruning methods, which are reported in Appendix A.8.

Notably, as illustrated in Table 1, even when rescaling is applied, magnitude-based pruning can sometimes impair the model merging performance. The comparison in Table 1 underscores that factors beyond rescaling - such as weight distribution - are also important for the merging performance, as further discussed in our response to Q2.

Q2: The impact of unbalanced weight distribution on merged model performance needs clearer evidence.

We provide supporting evidence in Table 4 in the main text. It compares different sparsification strategies for model merging: Task Arithmetic + Magnitude (layer-wise), + Magnitude (row-wise), and + BS (block-wise). As the pruning becomes more structured and the weight distribution more balanced, performance improves progressively from 80.38 to 80.61 to 81.30.

From a theoretical perspective, weight imbalance can amplify task interference during merging. Since the merged model is typically rescaled by a λ, if weights are highly concentrated in specific regions—as often caused by magnitude-based pruning—this rescaling will disproportionately amplify those regions, worsening cross-task interference.

While weight distribution may not be the sole cause, consistent improvements across balanced pruning provide strong empirical support. We acknowledge that deeper theoretical understanding, especially of activation dynamics, would be valuable. While our current work focuses on developing an effective, training-free merging method, we see this as a promising direction for future research.

Q3: The LLMs used for merging are too similar in performance; more diverse tasks are expected.

To address the concern about task diversity, we additionally conducted an experiment merging a Mistral model fine-tuned on instruction-following with another fine-tuned on mathematical tasks. As shown in Table 2 in anonymous link, our method still outperforms TaskArithmetic by +0.79, achieving SOTA performance in this heterogeneous setting and demonstrating robust cross-task merging capability.

We also would like to clarify that, merging models fine-tuned on similar tasks is a standard and widely adopted setup in model merging, especially under the model soup paradigm. Our large-model experiments follow DARE’s protocol, which focuses on improving performance or generalization within similar task domains.

This setup is also practically relevant: the Open LLM Leaderboard—one of the most active community benchmarks—frequently features submissions involving merges of LLMs fine-tuned on similar tasks. Using our method, we constructed four merged models and occupied the top four positions among all models with fewer than 8B parameters at the submission time (see Table 1 in the anonymous link or the official leaderboard).

Q4: The number of tasks/models in SLM experiments is limited; merging more models is recommended.

We appreciate the suggestion and have added an experiment merging all 7 GPT-2 models from FusionBench, following its official setup (see Table 7 in the anonymous link).

While CABS still achieves the best performance among all methods, the overall gains are notably smaller than in our 2- or 6-model merging settings. In fact, all methods—including CABS—suffer significant performance drops when merging more models, especially on some tasks.

We investigated this and found that the task vectors have highly imbalanced average weight magnitudes—up to a 10× difference across tasks (see Table 8 in the anonymous link). Since model merging uses a shared scaling factor (λ), tasks with small-magnitude vectors (e.g., RTE, MRPC, CoLA) are overwhelmed by stronger ones (e.g., QQP, MNLI), resulting in near-random performance on weaker tasks.

This suggests that blindly merging too many heterogeneous task vectors with vastly different magnitudes introduces severe imbalance and can degrade model utility. While CABS remains robust, merging fewer well-aligned models appears more practical and effective under current frameworks.

We hope our responses have addressed your concerns. Please let us know if there are any remaining questions or clarifications we can provide.

Thanks for your valuable feedback and constructive suggestions! Below, we address your main concerns:

Q1: Causality vs. Correlation in Experiments

We acknowledge the reviewer’s distinction between correlation and causation. While strict causal proof is difficult, our experiments go beyond correlation. In Figure 5, by fixing sparsity and method and varying only the overlap rate, we observe performance drop—indicating a causal link.

Table 4 shows that more structured pruning improves balance and performance (80.38 → 80.61 → 81.30). While full causality is not isolated, consistent trends across models highlight the practical value of addressing overlap and imbalance in training-free model merging.

Q2: why would pruning one then be any better?

Prior work shows that pruning improves model merging. TIES-Merging demonstrates that conflicting signs between task vectors cause interference, and pruning one side improves performance. DARE further shows that task vectors are highly redundant—keeping only ~5% of weights with proper rescaling still maintains near-lossless performance.

Unlike standard model pruning, task vector pruning is not for compression, but for reducing conflict and removing redundant updates that may interfere with other tasks—making it a strategic, empirically supported choice.

Q3: Why merge Task Vectors Instead of Learning a Joint Vector?

Task-vector-based merging aims to reuse independently fine-tuned models in a training-free manner, without access to original task data. In contrast, learning a joint vector requires access to all task data and additional training, which significantly increases computational cost.

CABS builds on TIES-Merging and DARE, proposing structured and conflict-aware pruning to enable efficient, training-free model merging.

Q4: Does CA over-prune task vectors by enforcing disjoint masks?

CA does not cause excessive pruning. In many cases, task vectors already reach 90% sparsity without CA. At the same sparsity level, CA improves merging by allocating disjoint weights, reducing interference while retaining essential updates.

Q5: If two tasks are fairly similar, why force them to orthogonality?

In our experiments with similar models—such as the model soup setting in Table 3 and Table 17 of the main paper, and Table 1 in Anonymous Link—we consistently observe that enforcing orthogonality via CABS still improves merging performance.

One geometric explanation is that, when task vectors are highly similar, their directions are nearly aligned, causing their linear combination to span only a narrow subspace. In contrast, enforcing orthogonality increases the span of the merged space. This broader coverage may facilitate better exploration of the solution space, enabling access to lower-loss regions.

Thus, even for similar tasks, CABS effectively improves expressiveness and reduces redundancy during the merging process.

Q6: Clarifying Eq. 7–9

Equation 7 defines a linear combination of two task vectors, but does not reveal how adjusting one scaling factor (e.g., λ_B) may interfere with the other task vector’s contribution in the resulting update.

For example, if τ_A = −τ_B, even small changes in λ_B can diminish the effect of τ_A, indicating strong coupling between the scaling factors. This illustrates that without further analysis, the two task vectors' contributions cannot be assumed independent.

The squared Frobenius norm in Eq. 8 explicitly captures this interaction via the inner product ⟨τ_A, τ_B⟩_F. As shown in Eq. 9, the cross term vanishes only when τ_A and τ_B are orthogonal.

This ensures that each λ independently controls its corresponding task vector’s contribution without unintended interference. Hence, the norm-based formulation is essential to formalize the decoupling effect of orthogonality.

Q7: Density vs Sparsity

We consistently use "sparsity" to refer to the percentage of pruned (zero) parameters—e.g., 90% sparsity means 90% of weights are zero. We do not use "density" in the paper, so there should be no ambiguity.

That said, we acknowledge that Table 2 does not explicitly mention the sparsity setting. However, the table references specific results in Tables 1, 8, and 9, where the sparsity level (90%) is clearly stated. We will revise Table 2 to include this information. We hope this clarification resolves the misunderstanding.

Q8: Why does rescaling work?

Rescaling is effective because task vectors are highly redundant—shown by DARE and our Figure 6 (Appendix A.8). Pruning up to 90%, with rescaling, yields near-lossless performance. Even at 99%, the drop is minimal, showing strong robustness.

Q9: The calibration dataset for SparseGPT and Wanda

We use C4 as the calibration dataset, following the standard setup in SparseGPT.

We hope these responses address your concerns. Please feel free to let us know if anything remains unclear.

Thanks for your valuable feedback and constructive suggestions! We've provided additional experimental tables in Anonymous Link. Below, we address your main concerns:

Q1: Inclusion of experiments on vision modality.

We added experiments on vision tasks (ViT-B-32 on DTD and EuroSAT, see Table 2 in the link). CABS surpasses Task Arithmetic by +2.20% average accuracy, confirming its effectiveness on vision tasks.

Q2: Justification of claims regarding outperforming the "ideal model".

We agree that this claim should be more cautiously presented. We will revise the paper to clarify the contribution statement and the discussion of Table 3 results, adding appropriate qualifiers to avoid overstatement.

Empirically, the claim is supported by consistent results across multiple settings (Tables 3, 11, 12, 14, and 15), including both Mistral and GPT-2 models and sparsity levels of 0.25 and 0.75 and 0.90. In particular, the improvement in GPT-2 experiments (Table 11) is free from randomness, as both the method (CABS) and the evaluation tasks (classification) are deterministic. As noted in Q4, confidence intervals in the large-model experiments further strengthen the reliability of the observed gains.

Q3: Clarification about the novelty of the "ideal model".

We agree that comparing to fine-tuned models is common for SLM, where the "ideal baseline" is equivalent to the ''fine-tuned models''. However, in LLM merging (e.g., multi-task LLMs), this type of upper-bound baseline is not used in prior large-model merging works such as DARE. We acknowledge that this distinction was not clearly explained and will revise the text accordingly.

Q4: Reporting confidence intervals.

Our magnitude-based merging method is deterministic for classification tasks, producing consistent results across runs. This aligns with common practice in model merging literature, where confidence intervals are often omitted. Still, for LLM evaluations, we conducted 10 runs:

These results confirm the statistical significance of our improvements.

Q5: Discussion of related work TALL-masks.

Compared to TALL-masks, CABS distinguishes itself in both design and applicability:

• TALL-masks assumes task id is available at inference, loading tuned task-specific masks—analogous to utilizing separate LoRA adapters—thus achieving near-lossless performance but requiring task-specific inference and added complexity.

• TALL-masks also involves tuning the mask sparsity factor for each task and scaling coefficients λ. While effective on small models, it has not been evaluated at LLM scale. In contrast, CABS only tunes λ and has demonstrated strong scalability: on the Open LLM Leaderboard, CABS occupied the top four positions among all sub-8B models at submission time (Table 1 in the link).

• TALL-masks produces a base model, a merged task vector, and k task-specific masks, resulting in >2× storage overhead. In contrast, CABS yields a single, compact model for multi-task inference.

Given these differences, direct comparison with TALL-masks is inappropriate. However, its merged variant Consensus TA is a fair baseline. We have added comparisons on merging 2,4,6 models (Table 4,5,6 in the anonymous link, Mask sparsity factor =0.5), with our results clearly demonstrating the superior performance of CABS. A discussion will be added in the Related Work section.

Q6: Effect of task ordering in CA.

We tested shuffled task orders explicitly (SLM: Table 1 in the main text; LLM: Appendix A.9). We further added results on merging six models with shuffled orders (Table 5 in the anonymous link). Results show that task ordering has limited impact relative to overall gains, demonstrating the robustness of the CA strategy.

Q7: Impact of overlap rates.

As discussed in the paper, the main issue is high overlap rather than all overlap. We agree some overlap can be beneficial—e.g., Figure 5 shows 20% sparsity achieving strong performance. However, determining the optimal overlap level introduces additional complexity. Considering the efficiency and generalizability, we adopt the current CA strategy.

Figure 5 is intended to analyze the effect of varying overlap rates within a fixed method (DARE), and already includes the result of applying CA with DARE. Adding CABS—which uses a different method (BS)—would conflate method differences with overlap effects.

Instead, we have updated Table 4 in the main paper (see Table 9 in the anonymous link) to enable a clearer comparison between BS and DARE under the same sparsity level.

Q8: Effectiveness on similar tasks.

We discuss this in our response to Reviewer 8hif (Q5).

We hope our responses have addressed your concerns. Please let us know if there are any remaining questions or clarifications we can provide.