BaWA: Automatic Optimizing Pruning Metric for Large Language Models with Balanced Weight and Activation

Dear zn1Z:

We sincerely appreciate the valuable suggestions provided by the reviewer. We note the two main concerns you raised, which we address below.

Firstly, we thank the reviewer for emphasizing the importance of comparing with structured pruning methods. We would like to compare structured and unstructured pruning in terms of pruning granularity, sparsity, accuracy, efficiency, and training after pruning. Structured pruning [1, 2, 3] removes complete substructures or weight groups (layers [4], FFN neurons [1], MHA heads [1], embedding dim [5], they are coarse-grained) from LLMs, enabling hardware-independent efficiency gains. However, under the constraints of this coarse-grained pruning, the post-pruned accuracy of LLM is prone to be drastically reduced, and therefore the generally applicable sparsity ratio is between 15% and 30%. Post-pruning fine-tuning/training/knowledge distillation may be used to restore the performance of the pruned model when faced with high sparsity ratios [2, 5]. Unstructured pruning removes elements from the weights (fine-grained) and stores or loads the pruned weights in a compressed assortment. Combined with decompression (memory bottleneck optimization) or hardware advantage (2:4 sparse tensor core), unstructured sparse LLM can also obtain considerable efficiency improvement. Unstructured pruning is less likely to harm the accuracy of the model due to the fine-grained constraints, so the sparsity can generally exceed 50%, and the adoption of a stricter 2:4 sparsity is also acceptable. It is possible to do unstructured pruning without loss of accuracy for large LLMs such as Llama-2-70B, and BaWA has done it. Unstructured pruned LLM can also be further trained, and some techniques such as PEFT [6] and STE [7] have emerged to provide support. In the revised manuscript, we will include a dedicated discussion (Section 2) comparing structured and unstructured pruning for LLMs to demonstrate the necessity of high-performance unstructured/semi-structured LLM pruning. The following table compares a portion of the experimental results of structured pruning and unstructured pruning as a reference.

Secondly, although unstructured pruning has traditionally lacked hardware support, recent advancements such as Flash-LLM [8] demonstrate its practical speedup through specialized kernels. Moreover, our method’s compatibility with N:M sparsity (natively supported by Ampere GPUs) further ensures deployability while maintaining higher accuracy than structured alternatives (Table 9). Additionally, we would like to clarify that the slight performance gap between BaWA and SparseGPT in semi-structured settings (Table 8) stems from SparseGPT’s weight reconstruction—a complementary technique orthogonal to pruning metrics, which is explained in our rebuttal to the reviewer xLm6. As shown in "BaWA+ADMM" (Table 3), combining our metric with weight reconstruction outperforms all baselines universally. Furthermore, larger models (e.g., LLaMA2-70B) exhibit greater robustness to N:M constraints due to inherent redundancy, reducing accuracy drops to <1% in 4:8 sparsity (Table 12).

Reference

[1] Llm-pruner: On the structural pruning of large language models, NeurIPS'23.

[2] Sheared llama: Accelerating language model pre-training via structured pruning, ICLR'24.

[3] Instruction-Following Pruning for Large Language Models, ArXiv'25.

[4] SlimGPT: Layer-wise Structured Pruning for Large Language Models, NeurIPS'24.

[5] Llm pruning and distillation in practice: The minitron approach, ArXiv'24.

[6] SPP: Sparsity-preserved parameter-efficient fine-tuning for large language models, ICML'24.

[7] Sparsity-accelerated training for large language models, ACL'24.

[8] Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity, VLDB'23.

Dear xLm6:

We greatly appreciate your insightful comments. Below we provide a point-by-point response to your concerns.

Scaling Factor Analysis：

We agree that analyzing scaling factors is critical. In the revised manuscript, we will add:

• A new comparative table (below) demonstrating the superiority of optimized scales over fixed values on LLaMA2-7B

• A novel discussion in the revised Section 4.4 will highlight how adaptive scaling addresses LLM-specific distribution challenges. Specifically, different models and task settings will be explored to illustrate the effectiveness of BaWA's scaling strategy.

The evaluation results can be shown as follows.

The key findings include:

• Optimized scales reduce perplexity by 12.3% compared to the best-fixed scale ($\theta$=0.5)

• Fixed scales exhibit significant sensitivity (±24.1% PPL variance)

Relationship with ADMM-Iter/DSnoT：

We apologize for the lack of explanation for the relationship between BaWA and ADMM-Iter/DSnoT. In fact, the LLM pruning procedure can be divided into two stages, pruning mask selection and weight reconstruction. Different from BaWA which optimizes pruning metric (Stage 1), these methods (both ADMM and DSnoT) focus on reconstructing post-pruning weights (Stage 2). To illustrate the orthogonality, we validate when adding weight reconstruction methods (both DSnoT and ADMM) with BaWA, as well as comparing them with SparseGPT, ADMM-Iter and DSnoT without BaWA pruning metric on various LLaMA models with 2:4 sparsity. The results (Table below) clearly depict that using BaWA pruning metric with weight reconstruction methods achieves the best pruning performance. Furthermore, our evaluation in Table 17 (Appendix G) also demonstrates the effectiveness of adding weight reconstruction to BaWA pruning metric. We will add this pipeline diagram to Appendix G.

Dear mDNk:

We sincerely appreciate your thoughtful feedback regarding BaWA’s novelty and computational overhead. Below, we address your concerns in detail.

Novelty of BaWA:

We respectfully disagree with the provided novelty concern for three key reasons:

(a) Problem Characterization in LLMs

• Structural Outliers: LLMs exhibit extremely sparse activation outliers (>50% outliers in <5% channels, Fig 1c), unlike CNNs with uniform noise patterns [1].
• Cross-Layer Imbalance: Weight magnitudes vary by 100× within layers (Fig 1a), violating CNN pruning assumptions.

(b) Methodological Advancements

• Dual-Channel Normalization: Joint input/output scaling (Eq. 5) handles asymmetric LLM distributions, unlike standard normalization.

• Dynamic Outlier Suppression: Learnable threshold $\theta_3$ adapts to layer-wise outlier density, improving upon fixed-threshold methods [2].

(c) Empirical Superiority

• BaWA reduces perplexity by 2.49 over Wanda (Table 3), which is a great improvement.

• Direct application of robust CNN pruning degrades accuracy by 4.1% (Table 8 in Appendix).

Computational Overhead:

Furthermore, the evaluation results in our paper have shown that the additional overhead of BaWA is negligible in two aspects.

(a) One-Time Search Cost: Searching a 70B model takes only 16 minutes (Table 6), <0.01% of typical training costs (thousands of GPU hours).

(b) Search-Free Mode: Even without search (BaWA w/o search), BaWA outperforms Wanda while maintaining the same pruning efficiency (Table 5).

References

[1] Li et al., Pruning Filters for Efficient ConvNets, ICLR 2017.

[2] Wei et al., Outlier Suppression+, EMNLP 2023.