Towards Efficient Adaptation of Pruning Strategy in Large Language Models

Thank you for your valuable feedback and for taking the time to review our work. We appreciate your recognition of our contributions.

---

Q1. Efficiency Testing Experiments.

We have conducted efficiency testing experiments, as detailed in Appendix A.3 of the paper. The results demonstrate the effectiveness of our proposed method in terms of both speed and resource utilization. We here present the table for your convenience.

Table 1: Cost of Searching for Optimal Pruning Metric and Layerwise Sparsity Ratios on LLaMA-1/2/3 and Mistral Models

As shown in the table, the time of an optimal search is within 2.5 GPU hours. With multiple GPUs, the search process can generally be finished within 1 hour. Therefore, we believe that the search cost of our method is moderate and acceptable in real-world scenarios.

---

Q2.Experiments at Other Compression Rates.

Our experiments also include evaluations at various compression rates, which are presented in Figure 3(a). These results illustrate how our method performs under different levels of sparsity ranging from 10% and 60%.

Table 2: Comparison of pruning metrics across different compression rates on WikiText perplexity.

The perplexity results consistently demonstrate that our Mecon method outperforms the baseline approaches across all the tested sparsity levels. Notably, at the challenging 60% sparsity ratio, Mecon exhibits a significant performance advantage. We hope this clarifies any oversight of our experiment results.

---

Q3. Additional Dataset Evaluations.

We have also evaluated our method on 7 datasets from LM-Harness to showcase its generalizability, including BoolQ, RTE, HellaSwag, WinoGrande, ARC-easy, ARC-challenge and OpenbookQA. These evaluations can be found in Section 4.2, Table 2 and Table 3, where we discuss the performance across different datasets from LM-Harness. Besides, we report the detailed task-wise accuracy on LM Harness in Appendix A.7.

---

Q4. Performance Improvements from Magnitude Pruning.

As you noted, magnitude pruning performs well on the LM-Harness tasks for the Mistral 7B model, even exceeding the performance of activation-considered pruning metrics like SparseGPT, Wanda, and RIA. The improvement seen with Mecon may appear marginal. However, Mecon's benefits extend beyond individual metrics; its adaptive approach maintains effectiveness across a broader range of models and tasks.

A deeper understanding lies in the performance differences between magnitude pruning and activation-considered pruning metrics across various models and tasks.

• Magnitude pruning evaluates weight importance solely based on absolute values, making it more robust to shifts in calibration data distribution, as it does not depend on specific data patterns for pruning decisions.

• Activation-considered pruning methods, like SparseGPT, Wanda, and RIA, assess weight importance based on activations relative to calibration data. If the calibration data differs significantly from the distribution during model pretraining, these methods may misestimate the importance of certain weights.

Consequently, when using the same calibration data across different models (e.g., LLaMA and Mistral), magnitude pruning and activation-considered methods can behave differently.

---

We hope our response addresses your concerns and provides clarity on the effectiveness of our proposed method. Thank you once again for your valuable feedback!

Thank you for your thoughtful feedback and for taking the time to review our paper. We sincerely appreciate your recognition of our work and would like to address your comments and questions individually.

---

Q1. Did the authors consider a more general representation to model the relationship between weights and activations?

Yes, we explored for a more general representation between weights and activations

In our analysis of the optimal searched pruning metrics, We find that the differences between transformed weights and transformed activations may affect the effectiveness of different pruning metrics. Specifically, we analyze each pruning metric, such as Wanda, RIA, and Mecon, by decomposing them into two distinct components: the transformed weights and the transformed activations, As the SparseGPT metric combines weights and the Hessian matrix, we omit the weight and activation analysis for SparseGPT.

We measure the difference as the layer-wise absolute difference, calculated by summing the average absolute differences between transformed weights and activations across all linear sub-modules in each layer. The average layer-wise differences are reported in Table 1, with detailed layer-wise difference curves provided in Appendix A.6 of our revised paper.

Table 1: Average absolute difference between operated weights and operated activations for Wanda, RIA and Mecon on C4 Calibration Data.

Table 1 shows that the RIA pruning metric reduces the absolute difference compared to Wanda, while the Mecon searched metric further minimizes this difference, bringing it close to zero. The weighted transformation operation in the Mecon pruning metric effectively scales both weights and activations into a similar numerical range.

Table 2: Mean zero-shot accuracies(%) on the LM Harness for Wanda, RIA and Mecon methods.

Coupled with the performance results of each pruning metric presented in Table 2, the difference analysis in Table 1 suggests that pruning metrics with smaller absolute differences between transformed weights and activations are more likely to achieve more effective pruning. Therefore, given this more general relationship between weights and activations, future pruning metrics could focus on reducing the output difference before and after pruning, as well as minimizing the difference between transformed weights and activations simultaneously.

---

Q2. Under Logic for Pruning Metric Search.

Our motivation for pruning metric search stems from our observations that the same pruning metric can yield significantly different performance across various models and tasks. For instance, we found that a simple magnitude-based pruning metric can outperform more complex metrics, such as SparseGPT, Wanda, and RIA, in certain models like Mistral 7B on commonsense tasks. Conversely, when applied to the LLaMA3 model, which has a different architecture and training strategy compared to LLaMA1 and LLaMA2, existing pruning metrics often lead to a substantial decline in performance. This variability drives our search for pruning metrics that are effective across diverse models and tasks.

Our meta pruning metric builds upon existing metrics derived from SparseGPT, with modifications made by Wanda and RIA to enhance their applicability. Theoretically, while SparseGPT approximates the Hessian matrix and uses Optimal Partial Updates to reduce computational costs, this approach limits error compensation due to fewer weights being available for adjustment. As a result, SparseGPT exhibits varying weight adjustment capabilities across different models with distinct weight distributions, leading to different performance outcomes.

To address these challenges without significantly slowing down the pruning process by considering weight distribution in the weight adjustment process, we design a lightweight pruning metric search method. We keep each search trial under 10 seconds on 7B models for its practical use.

---

Thank you for your thoughtful feedback and for taking the time to review our paper. We greatly appreciate your recognition of the novelty of our method, as well as your acknowledgment of our strong evaluation and clear writing.

---

Q1: Most Important Pruning Metrics.

To highlight the key metrics that consistently demonstrated effectiveness across models and tasks, we verify the generalizability of our Mecon-derived pruning metrics through cross-task and cross-model evaluations, showing that metrics developed for complex arithmetic reasoning tasks also perform well on simpler tasks like commonsense reasoning and language modeling, and remain effective when applied to models of different configurations. Thus, we could adopt the pruning metric identified on the challenging task with the strongest model in the candidate pool.

• As a result, for the LLaMA-1 and LLaMA-2 models, the most effective pruning metric is the one identified through the search process on LLaMA-2 models using GSM8K calibration data. $S_{ij} = \alpha |W_{ij}| \cdot \beta \||X_{j}\||_{2}^{1/2}$ where $\alpha$ represents F_norm coefficient and $\beta$ represents to_sum coefficient.

• For the LLaMA-3 and Mistral models, the most effective pruning metric is the one derived from the search conducted on LLaMA-3 models using the same GSM8K calibration data.

S_{ij} = \alpha |W_{ij}|^{2} \cdot \beta \||X_{j}\||_{2}^{1/2} $$ where $\alpha$ represents to_mean coefficient and $\beta$ represents to_sum coefficient. --- **Q2: Insights from the Optimal Searched Pruning Metrics.** Our meta pruning metric builds upon existing metrics derived from SparseGPT, with modifications made by Wanda and RIA to enhance their applicability. While SparseGPT approximates the Hessian matrix and uses Optimal Partial Updates to reduce computational costs, this approach limits error compensation due to fewer weights being available for adjustment. As a result, SparseGPT exhibits varying weight adjustment capabilities across different models with distinct weight distributions, leading to different performance outcomes. Moreover, in our analysis of the optimal searched pruning metrics, we find that the differences between transformed weights and transformed activations may affect the effectiveness of different pruning metrics. Specifically, we analyze each pruning metric, such as Wanda, RIA, and Mecon, by decomposing them into two distinct components: the transformed weights and the transformed activations, As the SparseGPT metric combines weights and the Hessian matrix, we omit the weight and activation analysis for SparseGPT. We measure the difference between transformed weights and activations as the layer-wise absolute difference, which is calculated by summing the average absolute differences across all linear sub-modules in each layer. We report the average layer-wise differences between the operated weights and the operated activations in Table 1, with detailed layer-wise difference curves available in our revised paper Appendix A.6. Table 1: Average absolute difference between operated weights and operated activations for Wanda, RIA and Mecon on C4 Calibration Data. | Method | L2-7B | L2-13B | L3-8B | Mistral 7B | |--------|--------|--------|--------|------------| | Wanda | 82.66 | 78.30 | 79.31 | 392.13 | | RIA | 22.34 | 21.91 | 21.15 | 44.77 | | Mecon | 1.09 | 0.0001 | 0.1263 | 0.0304 | Table 1 shows that the RIA pruning metric reduces the absolute difference compared to Wanda, while the Mecon searched metric further minimizes this difference, bringing it close to zero. The weighted transformation operation in the Mecon pruning metric effectively scales both weights and activations into a similar numerical range. Table 2: Mean zero-shot accuracies(%) on the LM Harness for Wanda, RIA and Mecon methods. | Method | L2-7B | L2-13B | L3-8B | Mistral 7B | |--------|-------|--------|-------|------------| | Wanda | 55.89 | 60.88 | 49.66 | 54.20 | | RIA | 55.67 | 61.03 | 50.76 | 54.39 | | Mecon | 57.47 | 61.42 | 52.48 | 59.33 | Coupled with the performance results of each pruning metric presented in Table 2, the difference analysis in Table 1 suggests that pruning metrics with smaller absolute differences between transformed weights and activations are more likely to achieve more effective pruning. Therefore, the performance of Wanda and other methods may be affected by their ability to account for these differences across various models with distinct weight magnitudes and distributions. --- We hope our explanation could address your concerns. Thank you again for your thoughtful feedback! We look forward to any additional comments you may have.

Thank you for your thoughtful feedback and for taking the time to review our paper. We greatly appreciate your recognition of the significance of our work. Below, we would like to address your comments and questions:

---

Q1. Under Logic for Pruning Metric Search.

Our motivation for pruning metric search stems from our observations that the same pruning metric can yield significantly different performance across various models and tasks. For instance, we found that a simple magnitude-based pruning metric can outperform more complex metrics, such as SparseGPT, Wanda, and RIA, in certain models like Mistral 7B on commonsense tasks. Conversely, when applied to the LLaMA3 model, which has a different architecture and training strategy compared to LLaMA1 and LLaMA2, existing pruning metrics often lead to a substantial decline in performance. This variability drives our search for pruning metrics that are effective across diverse models and tasks. We design a lightweight pruning metric search method, keeping each search trial under 10 seconds on 7B models for practical use.

---

Q2. Design of Meta Pruning Metric.

In developing our pruning metric, we drew inspiration from existing methods such as Wanda and RIA. We aimed to create a more effective metric by combining the absolute value of weights with the $L_2$ norm of activations, while also incorporating coefficient and operation adjustments as seen in RIA.

To clarify, our meta pruning metric simultaneously applies operations and coefficients to both weights and activations. The candidates for these operations were selected from common techniques used in expression searches, such as squaring, square roots, exponentiation, and logarithms. We also integrated various coefficients like the relative sum and row/column sums to enhance the metric's effectiveness.

---

Q3. Performance of Simple Coefficient Tuning.

Thank you for your insightful suggestion. We further conduct experiments to explore this aspect by tuning the coefficients associated with the Wanda pruning metric. We present our findings in the following tables.

Table 1. Comparison of coefficient tuning metric with Wanda, RIA, and Mecon on WikiText perplexity.

Table 2. Comparison of coefficient tuning metric with Wanda, RIA, and Mecon on LM Harness.

Our experimental results indicate that our Mecon pruning metrics consistently outperform those that rely solely on coefficient tuning. Specifically, while simple coefficient tuning shows improved performance over Wanda on LLaMA1-2 models and achieves comparable results to RIA, it does not achieve satisfactory results on the Mistral model. This suggests that, although simple coefficient tuning can identify optimal solutions more quickly, it cannot replicate the effectiveness of pruning metrics that integrate both coefficients and operations. The operations in the meta pruning metric are necessary.

In our analysis of the optimal searched pruning metrics, We find that the differences between transformed weights and transformed activations may affect the effectiveness of different pruning metrics. We provide a detailed analysis of this relationship in Appendix A.6 of our revised paper. An effective pruning metric likely requires transforming both weights and activations into a similar numerical range. Operations like squaring or taking square roots can more effectively reduce the differences between weights and activations compared to simply multiplying by coefficients. Thus, solely tuning the coefficients associated with the Wanda pruning metric may not yield satisfactory performance compared to Mecon.

---

We hope our response addresses your concerns and look forward to any further comments you may have. Thank you once again for your valuable feedback!

Thank you for your thoughtful feedback and for taking the time to review our paper. We truly appreciate your recognition of our work, and we have added the appropriate citation for the Non-dominated Sorting Genetic Algorithm III as suggested. Thank you for helping us improve our manuscript.

Q1. Improvement Difference between 7B and 70B models.

Regarding your question about the significant improvements observed in 7B models compared to the negligible improvements in 30B and 70B models, we believe this can be attributed to several factors:

• Sensitivity to Pruning: Smaller models, with fewer parameters and a simpler structure, are more sensitive to pruning. When significant connections are removed, they may lose critical connections for effective information flow. Consequently, our pruning metric shows greater improvement in the 7B models by preserving these important connections.

• Complexity and Redundancy in Larger Models: In contrast, larger models often have more complex architectures that enable them to learn richer feature representations. This complexity provides multiple pathways for information flow, enabling these models to adapt more easily when specific connections are removed, thus maintaining performance even after pruning. As a result, this inherent complexity can limit the potential improvements that pruning can achieve.

We appreciate your feedback and will further explore pruning strategies for larger models. Thank you once again for your valuable feedback!