Thank you for your very useful suggestions, and for engaging with us to improve our work. Please see below the response to your concerns, which include a number of new analyses, and please let us know if there is anything else we can do to further improve our paper.

Question 1: How sensitive is the sequential pruning order in CA to task difficulty or model performance? Would a different ordering strategy (e.g., based on task complexity) potentially yield better results?

Answer: Please refer to the "Common Question 2: What is the impact of different pruning orders on final performance?" , we observed minor differences across pruning sequences, the overall impact on performance was relatively small compared to the advantages of CA.

we have not yet explored ordering strategies based on factors such as task complexity or difficulty. This is indeed an interesting direction, and we appreciate your suggestion. Conducting experiments to determine how sequence selection can further optimize results is a promising avenue for future research.

Question 2: Could CABS be extended to merge models across different architectures or pre-training objectives?

Answer：Please refer to the "Common Question3: How does the method perform on cross-architecture merging?"

Question 3: In cases where sparsity levels exceed 1 and overlap cannot be completely eliminated, how do you determine which overlapping parameters to prioritize?

Answer: We have considered how to handle overlap when it cannot be fully eliminated. Our current approach, as described in Appendix A.3 (Algorithm of Low-Overlap Sparsity), is to prioritize the most important parameters by retaining those with the largest magnitude in overlapping regions. This heuristic is simple yet effective, ensuring that critical parameters are preserved.

While our current implementation opts for a straightforward strategy, we acknowledge that other strategies, such as exploring alternative metrics to guide parameter retention in overlapping regions, could further refine performance.

Question 4: What is the computational overhead of CABS compared to simpler merging approaches when scaling to larger models (>13B parameters)?

Answer: Please refer to "Common Question 4: Can you provide complete computational overhead analysis?" for a detailed discussion of computational complexity.

To summarize, the computational overhead of CABS scales linearly with model size due to its complexity of $O(k * N *log m)$, where $N$ is the number of parameters in the model, $k$ is the number of task vectors, and $m$ is the block size. As the model size increases, the merging overhead grows proportionally, making it efficient even for larger models (>13B parameters).

Weakness 1: Limited exploration of sparsity levels below 0.25 and above 0.75

Answer: In prior work, sparsity levels of 50%, 75%, and 90% are commonly used benchmarks for evaluating sparsification techniques. In our study, we introduced the 25% sparsity level specifically to discuss scenarios where overlap is inevitable, ensuring a more comprehensive analysis.

For smaller models, we conducted experiments at 90% sparsity to evaluate the impact of these levels, as shown in Table 1 of our paper. Additionally, we are currently running experiments with 90% sparsity on the 7B model. Once completed, the results will be made available in an added rebuttal.

Weakness 2 & 3: Please refer to our responses to Question 2 & 1.

Weakness 4: Experiments focused primarily on English language tasks, leaving questions about multilingual applicability.

Answer: While our study primarily focuses on English tasks to ensure comparability with prior work, we have expanded our experiments to include two Korean language tasks (kobest_copa and kobest_boolq) to explore the multilingual applicability of our method. The updated results are presented below:

It is important to note that for these tasks, we reused the merging configuration from our previous 7B experiments. This approach ensured consistency across evaluations and saved computational resources during the rebuttal period.

CABS achieves an average score of 75.41, closely matching the ideal model’s 75.59 (a difference of -0.18), while other methods fall significantly behind, with the best alternative, Task Arithmetic + DARE, achieving 74.63 (-0.96). This demonstrates that our approach maintains competitive performance across both English and non-English tasks.

We recognize that adding two Korean tasks, while helpful, may not fully address the reviewer's concerns about multilingual applicability. A more meaningful exploration would involve:

1. Models Fine-Tuned with Expanded Vocabularies: Multilingual models often expand their vocabularies during fine-tuning to enhance performance in specific languages. These models typically outperform those without expanded vocabularies in their respective languages but also introduce parameter dimension mismatches, making merging technically challenging for element-wise approaches.

2. Models with Native Multilingual Capabilities: Exploring models with large vocabularies and strong multilingual performance, such as those designed specifically for multilingual tasks, would further validate the method's applicability beyond English-centric datasets.

Due to the comparative and resource constraints of this rebuttal period, we could not explore these directions. However, we consider them promising avenues for future work and appreciate the reviewer's suggestion in highlighting this important aspect.

Question4: The score improvements in Tables 2 and 3 appear quite marginal compared to the baselines, and the results are not particularly compelling. Additionally, I believe it would be helpful to include the score of the pretrained model before fine-tuning to better understand the influence of the base model (W_base in Algorithm 1).

Answer: please refer to the ``Common Question7: Why do the improvements in Tables 2 and 3 seem small? ''

Question5: It would be helpful to present results from merging 3–5 models and compare them with the baselines to demonstrate the scalability and generalizability of the method.

Answer: please refer to the ``Common Question1: How would the method extend to merging multiple (>3) task vectors?''

Question 6: Could the authors provide guidelines on setting λ_A,B (the weighting constants of masked task vectors)? Are these values sensitive, and do they significantly impact performance?

Answer: The guidelines and sensitivity of λ_A,B have been discussed in Section 5.1 and Appendix A.9, where we analyze the impact of different grid intervals and explore the behavior of performance under varying $λ$ values. Below, we provide detailed explanations and experimental settings.

Guidelines for Setting $λ$:

• For small-scale tasks, we use a fine-grained grid search with an interval of 0.01 to ensure fair comparisons across methods and to avoid missing optimal values.
• For large-scale tasks (e.g., 7B models), we adopt a coarser interval of 0.1 due to computational cost considerations, consistent with prior work.

Experimental λ Values:

Large-Scale Models (7B)

Notes:

• For DARE at 0.75 sparsity, $λ = 2.2$ does not account for the sparsity ratio $1/(1-sparsity)$. If adjusted, this corresponds to $2.2/4 = 0.55$.
• The λ values for all methods, whether with or without TIES (e.g., TIES-DARE and DARE, TIES-Magnitude and Magnitude, TIES-CABS and CABS), are set identically for consistency.

Small-Scale Models (Sparsity = 0.90, Merge 4 Models)

Notes:

• For DARE at 0.90 sparsity, $λ = 4.61$ does not account for the sparsity ratio $1/(1-sparsity)$. If adjusted, this corresponds to $4.61/10 = 0.461$, similar in Ties-dare.

Sensitivity Analysis:

To investigate the sensitivity of λ, we conducted experiments and plotted performance across various λ values, which are presented in Appendix A.16.

1. Findings from Sensitivity Analysis:

   • Within a certain range, λ values are not particularly sensitive. For example, multiple λ-pairs achieve top-1 performance, indicating a basin-like structure in the loss landscape.
   • However, this range is not overly wide, emphasizing the importance of using finer intervals (e.g., 0.01) for smaller models to ensure accurate evaluation and avoid unfair comparisons caused by missed optimal values.

2. Reviewer-Specific Discussion:

   • Please refer to our response to Reviewer Goun for Weakness: Missing statistical significance analysis for Small models like RoBERTa, where we performed top-20 statistical analyses to evaluate method performance and λ sensitivity. This analysis can also demonstrates that different methods show varying levels of sensitivity to λ values. Notably, CABS exhibits relatively low sensitivity, highlighting one of its advantages.

3. Analysis in Related Work:

   • In Figure 15 of Task Arithmetic (https://arxiv.org/abs/2212.04089) and Figure 8 of TIES-Merging (https://arxiv.org/abs/2306.01708), also analyzed the influence of λ on performance, showing how λ impacts performance across different merging methods.

We hope these explanations, combined with the insights from our analyses, address the reviewer's concerns regarding λ guidelines and sensitivity.

Question7: Would it be possible to extend the experiments to include a generation-based benchmark, such as IFEval? I am curious about the model's generation capabilities and instruction-following ability after merging.

Answer: we add the result of IFEval task result in the table in ``Common Question7: Why do the improvements in Tables 2 and 3 seem small? ''.

Thank you for your very useful suggestions, and for engaging with us to improve our work. Please see below the response to your concerns, which include a number of new analyses, and please let us know if there is anything else we can do to further improve our paper.

Question1: Since sequential pruning is utilized, I believe the task order could impact performance. For instance, extracting the mask for task A and then using its inverse mask for task B might yield different results compared to extracting the mask for task B and using its inverse mask for task A. Could the authors analyze this aspect?

Answer: Please refer to the ``Common Question2: What is the impact of different pruning orders on final performance?''

Question2: Additionally, I am uncertain about the rationale behind the inverse mask strategy. When extracting the mask for task A and then using its inverse mask for task B, the inverse mask may remove important patterns for task B, as it is derived without consideration of task B. Could the authors provide a theoretical explanation and/or a more detailed analysis to justify this method?

Answer: We use this method caused by a Unique Properties of Task Vectors: Less Sensitivity to Parameter Selection

Task vectors are less sensitive to which specific parameters are retained compared to traditional model pruning. This unique property allows CA to focus on reducing overlap without significantly harming performance.

• Rescale experiments (Appendix A-14) demonstrate that even Random pruning—without any task-specific selection—can restore task vector performance to nearly the original level after rescaling. This highlights that task vectors are inherently less sensitive to the removal of specific parameters, making them more robust to pruning.
• In CA, there is a deliberate trade-off between retaining the most critical parameters and reducing overlap. Because task vectors are not highly sensitive to parameter selection, CA prioritizes reducing overlap, even if it involves sacrificing some critical weights. The sacrificed weights are typically less impactful, causing only negligible losses. By minimizing conflicts through reduced overlap, CA ultimately achieves better overall performance in merged models.

This robustness of task vectors allows CA to effectively balance minor sacrifices in critical weights with significant reductions in task interference, resulting in improved task independence and overall model performance.

Quantitative Validation of Overlap Reduction

Reducing parameter overlap directly mitigates conflicts between tasks, as shown in our experiments:

Magnitude pruning retains high-magnitude weights across tasks, causing significant overlap (71.42%) and task interference. CA, by contrast, assigns disjoint parameter regions to tasks, achieving 0.00% overlap and fostering task independence, resulting in improved accuracy. This empirical evidence validates CA’s approach to overlap reduction.

Question3: Magnitude and random pruning are commonly regarded as baselines in network compression literature. What might happen if alternative pruning criteria, such as the geometric median criterion , were used instead?

Answer: Please refer to ``Common Question 6: Why not choose other advanced pruning techniques?''

Thank you for your very useful suggestions, and for engaging with us to improve our work. Please see below the response to your concerns, which include a number of new analyses, and please let us know if there is anything else we can do to further improve our paper.

Weakness1: CA is simple, but questionable. However, what if you exchange the pruning order to task B first and then task A? The resulting model will not be equivalent, right? Also, what if you have > 2 models to merge? The later tasks will almost aways has less effective weights to be pruned from.

Answer: We have addressed both aspects of your concern in our responses to related Common questions:

1. The issue of merging more than two models is discussed in ``Common Question 1: How would the method extend to merging multiple (>3) task vectors?'', where we analyze how CABS performs with multiple task vectors. Our results demonstrate that even with >2 models, CABS consistently outperforms other methods. For example, while TIES-dare, the best-performing baseline, improves upon vanilla Task Arithmetic (TA) by +0.33, CABS achieves a strong average improvement of +2.15 over vanilla TA. This highlights that CABS is not only simple but also robust and effective across different merging scenarios.

2. The impact of pruning order is specifically addressed in ``Common Question 2: What is the impact of different pruning orders on final performance?''. While we observe small performance differences across different pruning orders (e.g., task A first vs. task B first), the differences are minimal relative to the overall improvements achieved. However, we also acknowledge the phenomenon you mentioned: "The later tasks will almost always have fewer effective weights to be pruned from."

We appreciate this observation and have discussed potential strategies to mitigate order-related issues. For example:

• Assigning larger λ values or higher densities (less pruning) to later task vectors.
• Performing multi-round pruning to balance the pruning impact across tasks.

These strategies represent promising directions to further refine CA and reduce order sensitivity, which we plan to explore in future work.

Weakness2: CA can be combine with any pruning technique, I do not see why BS here is a particularly good pruning option. n:m pruning is a paper from 2021, and there are multiple state-of-the-art pruning techniques recently. While the paper empirically proves BS is better than basic magnitude based pruning, what about other advanced pruning techniques. If CA + any pruning technique better than n:m pruning results in better performance, then we should not over-sell the importance of BS.

Answer: Please refer to the ``Common Question 6: Why not choose other advanced pruning techniques?''

Weakness3: The improvements are small compared to competing methods, what are the confidence intervals of these results?

Answer: Please refer to the ``Common Question 7: Why do the improvements in Tables 2 and 3 seem small?''

Weakness 4: Sparsification is only necessary for pruning-based model merging techniques. That said, I would like to see how the method compares against other categories of model merging techniques such as evolutionary model merge (https://arxiv.org/abs/2403.13187) and pack of llms (https://arxiv.org/abs/2404.11531).

Answer: The methods you mentioned, Evolutionary Model Merge and Pack of LLMs, differ significantly from pruning-based approaches like ours, particularly in terms of computational overhead during merging and inference. Below, we discuss the distinctions and trade-offs.

1. Methodological Focus and Overhead Differences

• Pruning-based methods like ours focus on efficiently reducing parameter conflicts during model merging without increasing computational and memory overhead during inference. This makes them ideal for scenarios where resource efficiency is critical.

• In contrast:

  • Evolutionary Model Merge relies on computationally expensive evolutionary algorithms to search for optimal layer-level combinations across models. While its merged models may achieve better absolute performance, the merging process has a significantly higher computational cost. Additionally, the resulting model sizes are often larger than the sum of the inputs (e.g., merging two 7B models could yield an 11B+ model), further increasing inference overhead.
  • Pack of LLMs, akin to an ensemble method, aggregates multiple models during inference without creating a unified merged model. While this allows cross-architecture integration, its inference costs scale linearly with the number of models, often requiring several times more computational resources compared to a single merged model. Furthermore, the method's merging process still involves searching for optimal logit weights, which introduces some computational overhead, less than merging techniques.

2. Practical Limitations

• Evolutionary Model Merge:

  • The method's code is not publicly available, and its computational requirements for merging make it challenging to reproduce or compare directly within the rebuttal timeframe.

• Pack of LLMs:

  • Tools like lm-evaluation-harness do not fully support ensemble evaluation. While we manually attempted to adapt these tools for evaluating Pack of LLMs on 7B models, the lack of robust support made direct comparisons infeasible within the rebuttal period.

Conclusion

Pruning-based methods like ours excel in scenarios where efficient merging and inference are critical, as they maintain the original model size and focus on resolving parameter conflicts. This stands in contrast to Evolutionary Model Merge, which prioritizes extensive optimization at a high merging cost, and Pack of LLMs, which emphasizes ensemble-style performance but with significantly higher inference costs.

We will expand the Related Work section in the revised paper to discuss these approaches and their trade-offs relative to pruning-based methods.

Question7: Can you provide complete computational and memory overhead analysis?

Answer: For computational overhead analysis，please refer to the ``Common Question 4: Can you provide complete computational overhead analysis?''. For memory overhead analysis：

We analyze the peak memory usage of different methods during the merging process and highlight that our method (CABS) does not introduce significant additional memory overhead:

1. DARE:
   • During merging, DARE requires loading both source models entirely into memory, resulting in a peak memory usage of O(2 * N * 2 bytes) (using lazy loading), where N is the number of parameters in a model.
2. TIES-Merging:
   • During the election phase, TIES-Merging requires loading all task vectors simultaneously, resulting in a peak memory usage of O(k * N * 2 bytes), where k is the number of task vectors.
   • If lazy loading is applied, the peak memory usage can be reduced to O(2 * N * 2 bytes).
3. CABS:
   • During the sorting phase, CABS requires memory for weights and two boolean-like masks: one to track weight usage and another to record pruning results. The peak memory usage in this phase is O(N * 2 bytes + 2 * N * 0.125 bytes).
   • During merging, the peak memory usage is the same as DARE and the optimized version of TIES-Merging, O(2 * N * 2 bytes) (using lazy loading).

The additional memory overhead introduced by CABS is very small, as merging is typically performed on CPUs. We have not encountered any memory bottlenecks, so optimizing for memory efficiency was not a priority. This analysis confirms that CABS remains highly memory-efficient and practical.

Question8: How does the method perform on cross-architecture merging?

Answer: Please refer to the ``Common Question 3: How does the method perform on cross-architecture merging?''.

Question9: How would the method perform on larger models (30B+)?

Answer: Please refer to the ``Common Question 5: How would the method perform on larger models (30B+)?''.

Question10: How does computational complexity scale with number of task vectors?

Answer: This concern is addressed in our response to a similar question: Can you provide complete computational and memory overhead analysis? In that response ( ``Common Question 4: Can you provide complete computational overhead analysis?''), we analyzed the computational complexity for merging when the number of task vectors is k. We believe reviewing that analysis will clarify how computational complexity scales with the number of task vectors in our approach.

Question2: How would the method extend to merging multiple (>3) task vectors?

Answer: please refer to ``common question 1: How would the method extend to merging multiple (>3) task vectors?''.

Question3: What is the impact of different pruning orders on final performance?

Answer: please refer to ``common question 2: What is the impact of different pruning orders on final performance?''.

Question4: How do you determine optimal n:m ratios for different model sizes?

Answer: We have discussed in Appendix A.13 (Table 10), where we examine how different n:m ratios affect the performance of the merged model while keeping the overall sparsity fixed at 75%. Specifically, we experimented with several n:m ratios, including 16:64, 32:128, 64:256, and others.

The results indicate that while higher n:m ratios (e.g., 32:128) tend to show slight improvements, the overall impact of varying n:m ratios is relatively subtle. This suggests that model performance is not highly sensitive to the choice of n:m ratios under the fixed sparsity constraint.

For our experiments, the optimal ratio was determined empirically by testing a few configurations, as shown in Table 10. Future work could explore more systematic methods for selecting these values, particularly for larger models or varying sparsity levels.

Question5: Why choose simple linear combination when there are more sophisticated approaches available, such as Fisher Information-based merging, gradient-based adaptive weighting, or attention-based parameter fusion? Have you considered incorporating these methods to potentially improve the merging performance?

Answer: Our decision to use a simple linear combination is driven by the following considerations:

1. Consistency with Prior Work: We aim to follow the methodology used in prior works such as Task Arithmetic (Ilharco et al., 2022) and TIES-Merging (Yadav et al., 2024), which also employ linear combination strategies. This ensures that our approach is directly comparable to these established baselines, making it easier to evaluate the improvements introduced by our method under similar conditions.

2. Scalability for Large Models: Methods like Fisher Information-based merging , gradient-based adaptive weighting, or attention-based parameter fusion often involve significantly higher computational overhead, such as additional backpropagation or Hessian calculations. These approaches, while potentially beneficial for smaller models, are less practical for large-scale models due to their computational and memory demands. For example, merging tasks in 7B or larger models requires methods that remain computationally efficient, a key factor in choosing a linear combination approach.

3. Focus of This Work: Our study primarily aims to address parameter overlap and its impact on merging performance, which is orthogonal to the choice of merging mechanics. By using a simple linear combination, we isolate the effect of our proposed low-overlap sparsity strategies without introducing additional complexity. Exploring more sophisticated merging techniques remains an exciting direction for future work.

References:

1. Ilharco G, Ribeiro M T, Wortsman M, et al. Editing models with task arithmetic[C]//The Eleventh International Conference on Learning Representations.
2. Yadav P, Tam D, Choshen L, et al. Ties-merging: Resolving interference when merging models[J]. Advances in Neural Information Processing Systems, 2024, 36.

Question6: Why weren't other major architectures (LLaMA, GPT) tested?

Answer: For larger models, we focused on the Mistral family, as it was the primary architecture used in DARE for large-scale merging experiments. This choice ensures consistency and enables direct comparisons with DARE’s results.

While DARE also conducted experiments with a LLaMA 2-based Wizard-Math-v0.1 model, this model was later withdrawn from Hugging Face by its authors. Despite our efforts to seek assistance via GitHub issues and emails, we were unable to access the model. As a result, we decided to focus on Mistral-based experiments, which were fully accessible and directly comparable to DARE’s primary results.

Although newer architectures like LLaMA 3 were available at the time, we attempted to incorporate LLaMA 3-related experiments during the rebuttal period. However, for smaller versions, we were unable to find suitable fine-tuned models on Hugging Face, and for larger versions, the time required to conduct new experiments exceeded the rebuttal period. Consequently, we were unable to include LLaMA 3 results.

We have identified suitable models and datasets for experiments on GPT-2, and the results will be included as an added rebuttal once the experiments are completed.

Thank you for your very useful suggestions, and for engaging with us to improve our work. Please see below the response to your concerns, which include a number of new analyses, and please let us know if there is anything else we can do to further improve our paper.

Question1: Why does reducing parameter overlap improve model merging performance?

Answer: We analyze why reducing parameter overlap improves model merging performance from two perspectives: 1) Insights from Parameter Partitioning in MTL, and 2) unique properties of task vectors, including their robustness to parameter removal. Additionally, we provide partial results from Table 5 in Section 5 of our paper to support this conclusion.

1. Insights from Parameter Partitioning in MTL

In multi-task learning (MTL), parameter partitioning reduces overlap between tasks, which has been shown to improve performance by minimizing task interference. This principle extends to model merging, especially when merging models with some degree of similarity, where reducing parameter overlap similarly enhances performance.

• Adashare (Sun et al., 2020): Learns sparse masks that partition parameters into shared and task-specific regions during training, ensuring that task-specific parameters are only updated by their corresponding tasks.
• Task Adaptive Parameter Sharing (TAPS) (Wallingford et al., 2022): Dynamically partitions parameters at the layer level, assigning distinct parts of each layer to different tasks. Parameters assigned to one task are not updated during the training of other tasks, maintaining task-relevant isolation.

These methods highlight how parameter partitioning during training reduces task overlap and mitigates interference, a principle directly applied by CA to improve model merging performance.

2. Unique Properties of Task Vectors: Less Sensitivity to Parameter Selection

Task vectors are less sensitive to which specific parameters are retained compared to traditional model pruning. This unique property allows CA to focus on reducing overlap without significantly harming performance.

• Rescale experiments (Appendix A-14) demonstrate that even Random pruning—without any task-specific selection—can restore task vector performance to nearly the original level after rescaling. This highlights that task vectors are inherently less sensitive to the removal of specific parameters, making them more robust to pruning.
• In CA, there is a deliberate trade-off between retaining the most critical parameters and reducing overlap. Because task vectors are not highly sensitive to parameter selection, CA prioritizes reducing overlap, even if it involves sacrificing some critical weights. The sacrificed weights are typically less impactful, causing only negligible losses. By minimizing conflicts through reduced overlap, CA ultimately achieves better overall performance in merged models.

This robustness of task vectors allows CA to effectively balance minor sacrifices in critical weights with significant reductions in task interference, resulting in improved task independence and overall model performance.

3. Quantitative Validation of Overlap Reduction

Reducing parameter overlap directly mitigates conflicts between tasks, as shown in our experiments:

Magnitude pruning retains high-magnitude weights across tasks, causing significant overlap (71.42%) and task interference. CA, by contrast, assigns disjoint parameter regions to tasks, achieving 0.00% overlap and fostering task independence, resulting in improved accuracy. This empirical evidence validates CA’s approach to overlap reduction.

References

1. Sun X, Panda R, Feris R, et al. Adashare: Learning what to share for efficient deep multi-task learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 8728-8740.

2. Wallingford M, Li H, Achille A, et al. Task adaptive parameter sharing for multi-task learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 7561-7570.

Weakness: Missing statistical significance analysis for Small models like RoBERTa

Answer: Due to the nature of classification tasks, it is not feasible to generate multiple results for a single configuration using methods like temperature control. Furthermore, as our approach is based on magnitude pruning, it inherently exhibits minimal stochasticity.

To address this limitation, we performed a statistical analysis using the top-20 results from different λ-pair searches rather than only reporting the top-1 result, as in the initial tables. This provides a more comprehensive view of model performance and ensures a near-equivalent statistical analysis.

The table below summarizes the 95% confidence intervals and mean values for the top-20 results across methods:

To further validate the statistical significance, we performed a t-test comparing the top-20 scores of CABS and TIES-DARE, the next best-performing method in the table. The results are as follows:

• t-statistic: 22.45
• p-value: $1.57 \times 10^{-23}$

The t-test confirms that the difference in performance between CABS and TIES-DARE is statistically significant, with a p-value far below the 0.05 threshold.

Thank you for your detailed responses and analyses. I appreciate your effort in addressing my concerns, but I still have the following concerns unresolved, so I maintain my original rating:

1. SparseGPT and WANDA Comparison: You noted that SparseGPT and WANDA perform similarly to magnitude pruning and do not address task vector conflicts. However, a direct experimental comparison with CABS in the context of model merging would strengthen your claims. Could you include such a comparison to better highlight CABS’s advantages?
2. DARE Experiments: There appears to be an inconsistency regarding DARE’s use of LLaMA 2-based models. DARE specifies utilizing multiple models for GSM8K testing, not exclusively Wizard-Math-v0.1. For example, DARE states: "We utilized publicly available mathematical reasoning models, including the MetaMath-llema-7B, MetaMath-7B, WizardMath-7B, and Abel-7B, all based on the LLaMA2-7B architecture." Why were LLaMA2-7B-based models excluded from your experiments? Could you clarify or revise this?
3. Model Merging vs. Generalization: Your response doesn’t fully address the balance between generalization performance and merging benefits. If SparseGPT or WANDA-pruned models show better generalization, in what scenarios would using CABS for merging be preferable to directly deploying these pruned models? Additional clarification on this point would be helpful.
4. Scalability and Complexity: While your results demonstrate robust performance for 4-task merging, scalability to more complex or heterogeneous tasks is only discussed theoretically. Including experimental results for larger models or more task vectors would significantly enhance the paper's practical relevance.
5. Theoretical Depth and Mechanism Analysis: Although empirical results and MTL insights support reducing parameter overlap, the explanation lacks rigorous theoretical analysis. Expanding on the mathematical basis for why reducing overlap improves performance would strengthen the scientific depth. Additionally, the mechanism through which overlap reduction mitigates task conflicts is not thoroughly dissected. Could you provide deeper theoretical insights or an analysis of how overlap reduction affects task vector interactions?

Question7: Why do the improvements in Tables 2 and 3 seem small?(Goun,qavE,so7i)

Answer: The improvements in Tables 2 and 3, while appearing small in absolute terms, are highly meaningful and significant. They highlight the ability of our method to further enhance model performance on benchmarks where fine-tuned models already achieve near-optimal results. Below, we address this concern in detail:

1. Limited Improvement Space on High-Performing 7B Models

The datasets evaluated in Tables 2 and 3 (e.g., ARC, MMLU, TruthfulQA) are well-established benchmarks, and fine-tuned 7B models already exhibit exceptional performance on these tasks. Consequently, achieving further improvements is both challenging and highly valuable.

For example, in the DARE paper (Section 4.3), the authors note that Supermario v2, a 7B merged model that ranked first on the Open LLM Leaderboard as of January 2024, achieved only a +0.20 improvement over the unmerged model. Similarly, in our experiments, DARE achieves comparable results, reinforcing that small absolute gains are common for 7B models.

Despite this constraint, our method demonstrates a substantial improvement. Compared to other methods in Tables 2 and 3, our approach achieves a maximum +0.64 improvement, significantly outperforming the baselines. Furthermore, our results exceed the ideal model, which we introduce as a strict benchmark representing the best possible task-specific performance derived from individual models.

reviewer so7i advised extending the experiments to include a generation-based benchmark, such as IFEval. We have included IFEval in our experiments under a sparsity setting of 0.75. The results are presented in the updated table：

With IFEval included, CABS achieves an average score of 72.22, extremely close to the ideal model’s 72.26, while other methods fall significantly behind. This further highlights the strength of our approach, as it maintains robust performance across diverse tasks

2. Statistical Validation of Improvement

To substantiate the significance of our improvements, we conducted statistical tests on 10 runs of experiments for both TIES-DARE and CABS, using a temperature of 1. The results are summarized below:

Additionally, a t-test comparing the two methods yielded the following results:

• t-statistic: -9.28
• p-value: $6.62 * 10^{-8}$

These results confirm that the performance gains achieved by CABS are statistically significant (p < 0.05) and robust across multiple runs.

For transparency, the raw data for these experiments are as follows:

TIES-DARE:
0.7608, 0.7618, 0.7607, 0.7583, 0.7596, 0.7596, 0.7603, 0.7618, 0.7615, 0.7614

CABS:
0.7660, 0.7652, 0.7631, 0.7651, 0.7640, 0.7641, 0.7652, 0.7650, 0.7648, 0.7643

Conclusion

While the absolute improvements may appear small due to the high baseline performance of 7B models, our results are both practically significant and statistically validated. The combination of robust experimental design, meaningful gains over baselines, and statistical rigor demonstrates the effectiveness of our approach.

Question 6: Why not choose other advanced pruning techniques? (qavE, so7i)

Answer:
The key difference between Balanced Sparsity (BS) and other advanced pruning techniques lies in their objectives. BS prioritizes maintaining a balanced distribution of task vectors to minimize conflicts during model merging, while traditional pruning methods focus on accurately identifying and removing "unimportant" weights to reduce redundancy. Ironically, traditional pruning methods, even when more accurate, can exacerbate conflicts in model merging by amplifying task interference.

While we experimented with advanced pruning techniques like SparseGPT[1] and WANDA[2], their results were comparable to magnitude pruning in our model merging tasks. Since these methods failed to address the critical need for balanced task vector allocation, we did not pursue further exploration. This highlights that simply focusing on redundancy reduction is insufficient for optimizing model merging performance.

Experimental Validation
Task vector sparsification fundamentally differs from traditional pruning, as discussed in Section 2 and Appendix A.4. Traditional pruning aims to optimize task-specific knowledge retention by eliminating redundancy, often showing better results when sparsity requirements are relaxed (e.g., layer-wise 50% sparsity vs. structured 2:4 pruning).

In contrast, task vector sparsification for model merging benefits from stricter sparsity requirements, as shown in Table 4 (Section 5.4). For example:

• Layer-wise sparsification performs worse than row-wise sparsification, which is outperformed by Balanced Sparsification (BS).
• BS minimizes task vector conflicts, prioritizing balance over redundancy reduction, which is critical for merging performance.

Furthermore, Appendix A.14 demonstrates that task vector knowledge is inherently distributed across substructures, making it less sensitive to weight removal. Even simple Random sparsification achieves near-full recovery of fine-tuning performance on target datasets, while Random pruning fails completely in traditional one-shot pruning scenarios.

Conclusion
The unique contribution of BS lies in its ability to maintain a balanced distribution of task vectors, which is critical for reducing conflicts in model merging. This explains our choice to use BS, despite its weaker performance in traditional pruning contexts. Unlike other advanced pruning techniques, which excel at retaining task-specific knowledge, BS is specifically tailored to address the challenges of task vector sparsification in model merging.

References:

[1] Frantar E, Alistarh D. SparseGPT: Massive language models can be accurately pruned in one-shot[C]//International Conference on Machine Learning. PMLR, 2023: 10323-10337.
[2] Sun M, Liu Z, Bair A, et al. A simple and effective pruning approach for large language models[J]. arXiv preprint arXiv:2306.11695, 2023.

Question3： How does the method perform on cross-architecture merging?（ Goun，so7i，jFhp）

Answer: Like other task vector-based approaches, our method is currently limited to models with identical architectures, as it relies on element-wise operations for merging weights. For models with different pre-training objectives, techniques like parameter alignment are often necessary to align weights into the same loss basin [1]. For models with different architectures, approaches such as knowledge distillation provide a potential solution, enabling the transfer of knowledge into a shared representation [2]. These methods are fundamentally different from our focus, which enhances the merging of multiple models with the same architecture and initialization.

References:

[1] Ainsworth S, Hayase J, Srinivasa S. Git Re-Basin: Merging Models modulo Permutation Symmetries[C]//The Eleventh International Conference on Learning Representations.

[2] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).

Question4： Can you provide complete computational overhead analysis?

Answer: Building on the computational complexity analysis provided in our paper (please refer to Section 4 and Appendix A.5), we further examine how complexity scales with the number of task vectors, $k$, focusing on layer-wise complexity, where $N$ represents the total number of parameters within a single layer.

Balanced Sparsification (BS) Complexity

In the CABS framework, Balanced Sparsification (BS) operates efficiently by dividing each layer's parameters into small, fixed-size blocks of $m$ parameters. Within each block:

• The top $n$ weights are selected based on magnitude, requiring a localized sorting operation with complexity $O(m \log m)$ per block.
• Given $N / m$ blocks in a layer, the total complexity for a single layer is $O(N \log m)$ for one task vector.

For $k$ task vectors, the total complexity becomes: $O(k \cdot N \log m)$. This is significantly more efficient than traditional magnitude pruning, which sorts all $N$ parameters globally and has a complexity of $O(k \cdot N \log N)$.

Conflict-Aware (CA) Strategy Complexity

The Conflict-Aware (CA) pruning strategy introduces minimal overhead. It operates sequentially for each task vector, ensuring non-overlapping pruned regions. This sequential application of masks aligns with standard frameworks like MergeKit and adds negligible additional computational cost.

Parallelization Potential

CABS is inherently parallelizable, as task vector processing can occur independently across layers or blocks. With sufficient hardware:

• The pruning complexity per block can reach $O(m \log m)$, enabling efficient large-scale model merging with multiple task vectors.

Even without full parallelization, the overall complexity remains $O(k \cdot N \log m)$, which is computationally efficient and scalable for merging models with multiple task vectors.

Question5：How would the method perform on larger models (30B+)?（Goun,jFhp）

Answer： We have not conducted experiments on larger models due to resource constraints, although merging with CABS is computationally efficient and can be performed on CPUs, as noted in our ``common question 4: Can you provide complete computational overhead analysis?'' on computational overhead. However, we currently lack the resources to conduct inference with larger models.

Though we can’t perform such experiments, we analyze that CABS would work well on larger models based on its current performance. Larger models typically exhibit more redundancy, allowing CABS’s conflict-aware (CA) pruning to reduce task interference more effectively with even less sacrifice of critical task-specific weights compared to smaller models. Furthermore, the issue of unbalanced weight distribution within task vectors persists in larger models due to the increased parameter space and task complexity, where CABS’s balanced sparsity (BS) mechanism is expected to address this imbalance effectively, further enhancing overall performance.