Activation-Guided Consensus Merging for Large Language Models

We sincerely appreciate the time and effort you put into reviewing our paper. We appreciate the insightful feedback and address your concerns as follows:

---

R4W1: MoE architecture

R4A1: We would kindly clarify that as we stated in the end of section 2, Mixture-of-Experts (MoE) merging strategies require fundamental architectural modifications, including the incorporation of expert routing mechanisms and gating networks, which extend beyond the scope of parameter-space merging techniques addressed in this work [1]. While MoE represents a related research direction, its dynamic expert selection mechanism and architectural adaptations fundamentally distinguish it from the static parameter merging approaches investigated herein. We would like to note that the baseline methods compared in this paper didn‘t consider MoE merging either.

[1] Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM. COLM 2024

R4W2: Other calibration dataset

R4A2: To alleviate the reviewers' concerns, we have also chosen the recently released LIMO[2] dataset as the calibration dataset for our experiments. LIMO challenges conventional wisdom in mathematical reasoning by showing that models can achieve superior performance with significantly less, yet higher quality, training data. In line with the settings applied in s1K, we conducted experiments using LIMO on GSM8K and Minerva Math, and the results are presented below:

Our method demonstrates good adaptability to the new calibration dataset and exhibits robustness.

[2] LIMO: Less is More for Reasoning. COLM 2025

R4W3: Comparison of system 1/2 discussion to previous work

R4A3: System 2 architectures are fine-tuned with extended thinking chains to promote deliberate analysis through iterative self-assessment, error mitigation, and verification, albeit facing challenges related to redundancy. In our work, the concept of System 2 follows [3,4,5], which is consistent with the notion that "System 2 delineates a cognitive mode distinguished by deliberate, analytical, and consciously reasoned processes" mentioned in "Visual agents as fast and slow thinkers." Furthermore, these works also align in their definition of System 1. We appreciate the reviewer's reminder, providing us with a visual perspective on system 1/2. We will cite this paper in our related works section in the final version.

[3] OpenAI. Openai o3-mini: Pushing the frontier of cost-effective reasoning. Arxiv 2025.01

[4] Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Arxiv 2025.01

[5] From system 1 to system 2: A survey of reasoning large language models. Arxiv 2025.02

R4Q1&Q2-a: Usage of Mutual Information

R4A4: We present our rationale for utilizing mutual information from the following perspectives: 1) Definition and Application: Mutual information quantifies the interdependence between two variables, reflecting the extent to which knowledge of one reduces uncertainty about the other. It is extensively employed in representation learning [6]. In the context of model merging, as outlined in Section 3.1, mutual information assesses the relationship between the weights of the FT and PT models. High mutual information indicates redundancy (e.g., both models excel at similar tasks), resulting in limited performance improvements and potential noise introduction. Conversely, low mutual information suggests orthogonal knowledge, facilitating the effective integration of complementary capabilities and enhancing generalization. 2) Evaluation of Alternative Metrics: In our preliminary experiments, we considered KL divergence and cosine similarity to measure model relationships.

• KL divergence is asymmetric; for instance, with P=[0.9,0.1] and Q=[0.5,0.5], we found $D_{KL}(P∥Q)≈0.368$ and $D_{KL}​(Q∥P)≈0.511$, leading us to exclude this metric.
• Cosine similarity exhibits substantial variability across layers, approximately ranging from 0.25 to 0.8, which can negatively impact performance, as prior studies emphasized that weight coefficient fluctuations should be minimal [7]. Furthermore, the range of cosine similarity is (-1, 1), complicating the interpretation of negative values.

In summary, we selected mutual information as a robust tool for examining model relationships. Due to space limitations, we emphasized its relevance in model merging and our further experiments validate its effectiveness. We appreciate the reviewers' suggestions and will incorporate preliminary observations in the final version.

[6] A Mutual Information Maximization Perspective of Language Representation Learning. ICLR 2020

[7] Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models. ACL 2024

R4Q2-b: Illustration of ACM on “conciseness vs. verbosity”

R4A5: In the design of our ACM framework, when merging models with substantially different response patterns—such as in L2S tasks—or across heterogeneous tasks like code generation and mathematical reasoning, we assign relatively lower merging weights when the mutual information between corresponding layers is high, and higher weights when the mutual information is low. This adaptive weighting scheme effectively emphasizes task-specific (non-shared) features while preserving shared knowledge, thereby enhancing the model's ability to generalize across diverse task domains. We would kindly clarify that, as we mentioned in section 4.1 “Main Results”，we observed that response length positively correlates with question difficulty. Furthermore, as shown in the Appendix A.3, the merged model retains reflective capabilities; however, reflection frequency has decreased due to the PT fast-thinking model. According to our careful case study, while the merged model maintains favorable reasoning ability, it avoids redundant reflection on simpler mathematical problems, such as those in GSM8K, thereby reducing response length.

R4Q3: Analysis and Explanations of Gains on ACM

R4A6: In the L2S task, for instance, we would kindly clarify that ACM improves on TIES and R1 by 4.2% and 2.3%, respectively. Our manuscript shows ACM significantly outperforms AIM on the 1.5B model, with a supplementary 14B AIM experiment yielding 75.5% accuracy compared to ACM's 77.1% (a 1.6% increase). We will incorporate the AIM results into Appendix A.2..

Meanwhile, we would like to remind that as clarified in section 2, L2S task prioritizes length reduction while maintaining accuracy [8,9]; ACM achieves a 73.5% length reduction relative to R1 on the 1.5B model with minimal resources.

It can be observed that our model merging approach ACM is more efficient, with higher performance and lower length.

We report the average scores across five runs with different random seeds. In Section 5, a comparison is made between our method and reinforcement learning (RL) approaches. Despite the considerably higher training costs associated with RL, our method attains a comparable reduction in length.

[8] O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning. Arxiv 2025/01

[9] L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. Arxiv 2025/03

R4Q4: Hyperparameter and model size limitations

R4A7: In comparison to alternative approaches, our method requires searching only one hyperparameter, whereas other baseline methods typically necessitate multiple hyperparameter searches. Due to resource constraints, we were unable to conduct experiments on 100B-scale models; however, we have successfully performed experiments on 32B (mentioned section 4.1 and Appendix Table 5) and 1.5B (Page 7, Table 2) models as reported.

We would like to reiterate our sincere appreciation for the reviewer’s helpful and thoughtful comments.

We sincerely appreciate the valuable advice from the reviewer. And we would like to address your concerns as follows:

---

R3W1: Calibration dataset

R3A1: In Section 4 “Models and Datasets”, we mentioned that we cluster the data in s1K and uniformly sample 10% (1000 * 10% = 100 pieces) of the data for multiple experiments. This approach is intended to mitigate potential biases arising from uneven sampling. Additionally, in earlier experiments, we sampled varying sizes of calibration datasets for analysis, and the test results for GSM8K on the 7B model are presented below:

Our observations indicate that once the number of calibration data points reaches 100, accuracy remains high while length remains short. We also analyzed the weight coefficients at this threshold. When the number of data is below 100, the distribution of weight coefficients fluctuates significantly, with the top five layers showing considerable variation. Conversely, when the count exceeds 100, these characteristics stabilize. This trend is similar across models such as 1.5B and 14B. Thus, we advocate for a 10% sampling rate in our experimental design. For larger models (e.g., those exceeding 100B), additional calibration data may be required; however, resource limitations prevented us from further experimentation. We posit that activation-based methods necessitate fewer data and computational resources compared to supervised fine-tuning or reinforcement learning.

We also present the results of the 7B model using different sampling seeds as follows:

It is evident that our clustering and sampling methods are both reasonable and robust.

Additionally, we investigated the effects of varying the number of clusters. Our results demonstrate that random clustering results in suboptimal performance, indicating possible imbalance in the dataset's distribution. Notably, with and beyond 20 clusters, we observed improved accuracy and reduced length, suggesting that clustering then sampling effectively mitigates data imbalance.

We sincerely appreciate the reviewers' suggestions and will incorporate these experiments into the appendix.

R3W2: Computational overhead as a plug-and-play approach

R3A2: According to our analysis, the ACM method demonstrates exceptional computational efficiency on the experimental equipment mentioned in the paper: the 1.5B model experiment requires only about 40 seconds on CPU, while the 7B and 14B models need only about 1-1.5 minutes. Considering the significant accuracy improvement brought by ACM (such as 2.3% improvement over R1 for the 1.5B model) and the substantial length reduction (73.5% reduction compared to R1), the required computational overhead is negligible. Furthermore, some baseline methods, such as Sens-Merging and AIM, also require additional calibration calculations. As analyzed in section 5, Figure 5, compared to resource-intensive training methods, ACM can easily achieve significant length reduction effects at extremely low cost.

We further supplement the results with fine-tuning of the DeepSeek-R1-1.5B model (optimized for slow thinking) on s1K's COT data (optimized for fast thinking), which took approximately 10 minutes for three epochs on a GPU. The experimental results are presented below:

It can be observed that our model merging approach ACM is more efficient, with higher performance and lower length.

R3W3: Extension Experiments

R3A3: For a fair comparison, we followed the task selection of the baseline methods and conducted experiments on the L2S tasks (including both simple and complex reasoning tasks, code generation tasks), as well as general model merging tasks (math and code model merging tasks). We appreciate the reviewer's suggestion; due to limitations in API resources, we selected the IFEval dataset [2]—a benchmark for instruction following—to evaluate our method. The experimental results are as follows:

We find that on the instruction-following dataset, ACM can further enhance the model merging performance of TIES, thereby further validating the effectiveness of our ACM approach.

R3Q1: Mutual information on heterogeneous scenarios

R3A4: Mutual information quantifies the interdependence between two random variables, reflecting the extent to which knowledge of one reduces uncertainty about the other. It is extensively employed in representation learning [1]. In the context of model merging, as outlined in Section 3.1, mutual information assesses the relationship between the weights of the FT and PT models. High mutual information indicates redundancy (e.g., both models excel at similar tasks), resulting in limited performance improvements and potential noise introduction. Conversely, low mutual information suggests orthogonal knowledge (e.g., code and math tasks), facilitating the effective integration of complementary capabilities and enhancing generalization.

Therefore, in the design of our ACM framework, when merging models with substantially different response patterns—such as in L2S tasks—or across heterogeneous tasks like code generation and mathematical reasoning, we assign relatively lower merging weights when the mutual information between corresponding layers is high, and higher weights when the mutual information is low. This adaptive weighting scheme effectively emphasizes task-specific (non-shared) features while preserving shared knowledge, thereby enhancing the model's ability to generalize across diverse task domains.

We sincerely thank the reviewer for the insightful and constructive feedback. We hope our responses could adequately addressed the reviewer's concerns.

References:

[1] A Mutual Information Maximization Perspective of Language Representation Learning. ICLR 2020

[2] Instruction-Following Evaluation for Large Language Models. Arxiv 2023.11

We sincerely thank the reviewer for the constructive suggestions. We would like to address your concerns as follows:

---

R2W1&Q3-a&Q4: Analysis and Explanations of Gains on ACM

R2A1: In the L2S task, we would kindly clarify that ACM improves on TIES and R1 by 4.2% and 2.3% respectively, not 1% as the reviewer stated. Our Table 2 in the manuscript shows ACM significantly outperforms AIM on the 1.5B model, with a supplementary 14B AIM experiment yielding 75.5% accuracy compared to ACM's 77.1% (a 1.6% increase). We will incorporate the AIM results into Appendix A.2.

Meanwhile, we would like to remind that as clarified in section 2, L2S task prioritizes length reduction while maintaining accuracy[2,3]. It is expected that responses are slightly longer on the L2S task due to the integration of long CoT model, compared to short CoT model. ACM achieves a 73.5% length reduction relative to R1 on the 1.5B model with minimal resources.

As for resource consumption, as outlined in Section 4.1 Table 1, we employed merely 10% of the calibration data. In our experiments, merging a 1.5B model on a CPU required approximately 40 seconds, whereas fine-tuning the R1-1.5B model (optimized for slow thinking) with COT data (optimized for fast thinking) from S1K took around 10 minutes for three epochs on GPU.

The model merging approach proves to be more efficient, delivering higher performance with reduced length. In Section 4, we highlighted the DPO performance on the 7B model, demonstrating that model merging is both resource-efficient and effective. Section 5 compares our method to popular RL L2S approaches. Despite the higher training costs of RL, our method achieves a similar reduction in length. We report average scores across five runs with different random seeds.

In general tasks, smaller gains align with UniTE[1]'s recommendation that model accuracy differences should be less than 15%, as the performance disparity among models in the pool is substantial, resulting in limited improvements.

[1] Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling. ICLR 2025

[2] O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning. Arxiv 2025/01

[3] L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. Arxiv 2025/03

R2W2: Difference compared to Sens-Merging

R2A2: ACM and Sens-Merging are fundamentally distinct in methodology and design. 1) ACM leverages activation information—commonly used in model quantization [4]—and requires only forward propagation, whereas Sens-Merging depends on gradient computation, necessitating both forward and backward passes. This results in higher computational overhead for Sens-Merging. 2) ACM assesses model information correlation through mutual information, whereas Sens-Merging requires multiple task-specific and cross-task scaling hyperparameters, adding complexity and sensitivity. In contrast, ACM depends on a single hyperparameter, demonstrating robustness and simplicity. 3) Experimental results demonstrate that ACM consistently outperforms Sens-Merging, validating its effectiveness. Due to space constraints, we focused on ACM principles in the manuscript but will incorporate more details on baseline methods like Sens-Merging in the final version.

[4] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys 2024

R2W3: Usage of Mutual Information

R2A3: We present our rationale for utilizing mutual information from the following perspectives:

1) Definition and Application: Mutual information(MI) quantifies the interdependence between two variables, reflecting the extent to which knowledge of one reduces uncertainty about the other. It is extensively employed in representation learning [5]. In model merging, as outlined in Section 3.1, MI assesses the relationship between the weights of the FT and PT models. High MI indicates redundancy (e.g., both models excel at similar tasks or response styles), resulting in limited performance improvements and potential noise introduction. Conversely, low MI suggests orthogonal knowledge, facilitating the effective integration of complementary capabilities and enhancing generalization.

2) Evaluation of Alternative Metrics: In our preliminary experiments, we considered KL divergence and cosine similarity to measure model relationships.

• KL divergence is asymmetric; for instance, with P=[0.9,0.1] and Q=[0.5,0.5], we found $D_{KL}(P∥Q)≈0.368$ and $D_{KL}​(Q∥P)≈0.511$, leading us to exclude this metric.
• Cosine similarity exhibits substantial variability across layers, approximately ranging from 0.25 to 0.8, which can negatively impact performance, as prior studies emphasized that weight coefficient fluctuations should be minimal [6]. Furthermore, the range of cosine similarity is (-1, 1), complicating the interpretation of negative values.

We chose MI as a robust tool for examining model relationships. Due to space limitations, we emphasized its relevance in model merging and our further experiments validate its effectiveness. We appreciate the reviewers' advice and will incorporate preliminary observations in the final version.

[5] A Mutual Information Maximization Perspective of Language Representation Learning. ICLR 2020

[6] Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models. ACL 2024

R2W4: Activation Theoretical Analysis

R2A4: We sincerely thank the reviewer for this insightful comment. Our intention was to use ReLU as an illustrative example to clearly present the core mechanism of our proposed method. In our analysis, the central quantity of interest is $S(W) := sup_x sup_∇ ⟨∇, Δ W x⟩$, where the supremum is taken over the set of all possible gradients ∇ of the activation function. For this quantity to be well-defined and serve as a meaningful bound, the essential requirement is that the set of gradients must be bounded.

• In the specific case of ReLU, the set of gradients is simply {0, 1}, making the characterization of $S(W)$ very straightforward. We used this case to build intuition.
• For more general activation functions, such as GELU and SwiGELU, the gradient ∇ is not binary but belongs to a continuous, bounded interval (approximately [-0.17, 1.17]). Because this set is bounded, the sup_∇ operation is still well-defined, and our formulation of $S(W)$ remains valid and finite.

Therefore, our core argument does not rely on the gradient being binary, but on the fact that for almost all modern activation functions, the gradient is bounded. This ensures that $S(W)$ is a meaningful quantity that can be analyzed.

We appreciate the reviewer's feedback and will revise this section to introduce the quantity $S(W)$ and a bounded gradient. We will adjust space format as well.

R2Q1: Explanation of L100

R2A5: The task vector is defined as $δ_i = θ_i − θ_0$, $θ_i$ usually refers to the weights of fine-tuned models, and $θ_0$ is that of the pretrained model. Task vectors highlight fine-tuned features. Arithmetic merging, including average or weighted averaging (TA), directly operates on these task vectors.

R2Q2&Q3-b: Calibration dataset

R2A6: In Section 4 “Models and Datasets”, we mentioned that we cluster the data in s1K and uniformly sample 10% (100 pieces) of the data for multiple experiments. This approach is intended to mitigate potential biases arising from uneven sampling. Additionally, in earlier experiments, we sampled varying sizes of data for analysis, and the test results for GSM8K on the 7B model are presented below:

Our results show that accuracy stabilizes and response length remains short once calibration data reaches 100 samples. We analyzed the weight coefficients at this threshold. Below 100 data points, the distribution fluctuates significantly, particularly in the top five layers. However, when the count exceeds 100, these characteristics stabilize. This trend holds across 1.5B and 14B models, supporting our use of a 10% sampling rate. For larger models (e.g., >100B), more data may be needed, though resource constraints limited further study. We posit that activation-based methods necessitate fewer data and computational resources compared to SFT or RL.

We also present the results of the 7B model using different seeds as follows:

It is evident that our clustering and sampling methods are both reasonable and robust. We sincerely appreciate the reviewers' suggestions and will incorporate these experiments into the appendix.

Again, we sincerely appreciate the valuable suggestions provided by the reviewers. If our explanations can assist the reviewers in better understanding our paper, we would be delighted. Should the reviewers have any further questions, we welcome further discussion.

Thanks for the reviewers' feedback. In response to the remaining questions, we offer the following clarifications:

1. The reviewers noted that TIES demonstrates better length reduction in the 1.5B L2S model. While we acknowledge this, it's important to assess method effectiveness by considering both accuracy and length across different model sizes, such as 7B and 14B. In the 7B model, ACM achieves higher accuracy than Ties, with length reductions of 55.3% and 53%, respectively. For the 1.5B model, ACM exhibits greater length reduction on challenging datasets like AIME24, Olympiad Bench, and College Math compared to TIES. We emphasize that ACM effectively balances accuracy and length on the L2S task.

   ACM, as a low-cost method, can be effectively combined with approaches such as TA and TIES to enhance performance. If users of the 1.5B L2S prioritize both length reduction and performance, we recommend ACM. For those focused solely on maximum length compression, TIES may be a more suitable option. We hope the reviewers will assess ACM's effectiveness based on comprehensive experimental results.

2. Regarding the LLaMA experiments, we explained in our rebuttal that the significant accuracy difference between MammoMATH (41.55) and Codellama-7B (25.47) limits performance improvements after merging [1][2]. Besides, similar marginal improvements are observed in broader experiments like Sens-Merging and AIM-Merging, implying some common magnitude of model merging. We refer to the original texts of AIM [3] and Sens-Merging for further details.

Although our model versions differ from those used in Sens-Merging and AIM-Merging, the phenomenon of limited improvement is commonly observed. Given the variability in the availability of open-source models, the performance disparity of the models we are working with is even greater, which underscores the effectiveness of the ACM method.

3. We explained that ACM only requires a forward pass, whereas Sens-Merging involves both forward and backward processes, making the computation for Sens-Merging more complex. In terms of time overhead, in our experiments and replication process, ACM takes approximately 40 seconds on the 1.5B model, while Sens-Merging takes about 90 seconds.

[1] Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling. EMNLP 2024

[2] Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling. ICLR 2024

[3] Activation-Informed Merging of Large Language Models. Arxiv 2025.

We greatly appreciate the reviewer’s helpful feedback and recognition of our contributions. We would like to address your concerns as follows:

---

R1Q1: Availability of fine-tuning data

R1A1: We address the reviewers' considerations as follows:

• Data Acquisition Challenges: For fine-tuned models, the associated fine-tuning data is often inaccessible due to concerns regarding data privacy, ownership, and compliance with regulatory policies. For instance, while the HuggingFace platform hosts a large number of open-weights models, such as Qwen series and DeepSeek R1 series, the corresponding fine-tuning datasets are frequently unavailable or undisclosed.
• Orthogonality of Model Merging and Fine-Tuning: Model merging and fine-tuning are orthogonal processes. After fine-tuning, model merging can still be performed. For example, we can merge models that have been fine-tuned in specific domains, such as fast-thinking models (Qwen series) and slow-thinking models (R1-Distill series), thereby integrating their capabilities, obtain a model that possesses long-chain reasoning capability while maintaining concise overall responses.
• Comparative Time Efficiency: Fine-tuning is comparatively more time-intensive than model merging. In our experiments, merging a 1.5B model on CPU required approximately 40 seconds, whereas fine-tuning the DeepSeek-R1-1.5B model (optimized for slow thinking) with COT data (optimized for fast thinking) from s1K took around 10 minutes for three epochs on GPU. The experimental results are presented below:

It can be observed that the model merging approach is more efficient, with higher performance and shorter length. Additionally, we reported the performance of DPO on the 7B model in Section 4, Table 1, finding that model merging is a resource-efficient yet highly effective method.

Once again, we greatly appreciate the reviewer’s valuable feedback and constructive suggestions.