Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization

Dear Reviewer DF7P:

We greatly appreciate your insightful and constructive feedback. Please find our point-by-point responses to your concerns and questions below.

---

Weakness 1: "There are instances of carelessness in writing. For example, in line 71, the phrase “search the optimal sparsity distribution” is used, but the final letter ‘n’ in “distribution” is not bolded consistently with the rest of the word."

Response to Weakness 1: Thank you for pointing this out. We will correct the formatting inconsistency and carefully proofread the manuscript to eliminate similar and other typographical issues throughout the paper.

---

Weakness 2: "The search process incurs significantly higher computational cost compared to other structured pruning approaches such as LLM-Pruner and SliceGPT. Although the authors acknowledge this limitation and mention plans to improve efficiency in future work, the current method may be less practical in resource-constrained environments due to the heavy search overhead."

Response to Weakness 2: We acknowledge that there exists the cost of search in our method compared to some fast pruning approaches such as LLM-Pruner and SliceGPT. However, these methods often sacrifice accuracy and rely on costly post-pruning fine-tuning to recover performance. For example, as shown in Table 2, pruning 50% of LLaMA-3.1-70B results in SliceGPT and LLM-Pruner retaining only 62% and 45% of the dense model’s accuracy, respectively, whereas our method preserves up to 97%. For the former methods, substantial fine-tuning or knowledge distillation is typically required after pruning to restore accuracy—introducing additional computation and time overhead that can outweigh the initial pruning efficiency. Our method avoids gradient backpropagation during pruning (c.f., line 158, Section 2.4) and adopts an optimized supernet storage strategy (c.f., Appendix A.4), making the primary overhead come from search time rather than resource costs. The entire pruning process can be executed on only a single GPU for an 8B model, which ensures its applicability in resource-constrained environments. Overall, we believe that spending additional time to obtain a substantially higher-quality pruned model is a worthwhile trade-off, particularly in scenarios where inference efficiency and accuracy are both critical.

---

Question 1: "In Equation (3), the Taylor expansion of the error function $E=|\mathbf{X} \mathbf{W}-\mathbf{X} \widehat{\mathbf{W}}|_2^2$ is expressed in terms of derivatives with respect to $\mathbf{W}$. However, $\mathbf{W}$ is fixed and the actual variable is either $\delta \mathbf{W}$ or equivalently $\hat{\mathbf{W}}$. Could the authors clarify this?"

Response to Question 1: Thank you for the thoughtful question. While it is true that $\widehat{\mathbf{W}}$ is the optimization variable, our analysis adopts a perturbation-based perspective, where the focus is on how small deviations $\delta \mathbf{W} = \widehat{\mathbf{W}} - \mathbf{W}$ around a fixed reference point $\mathbf{W}$ influence the error function. The Taylor expansion is thus naturally formulated around $\mathbf{W}$. This approach is standard in model compression literature [1], where the goal is to assess the impact of weight perturbations. We will clarify this interpretation in the future version for better readability.

---

Question 2: "Equation (6) introduces two key hyperparameters, $\alpha$ and $\beta$. How sensitive is the final performance to different choices of them? Can the authors provide empirical results or practical guidance on setting these parameters?"

Response to Question 2: We express Equation (6) in a general form where the hyperparameter $\alpha_l$ controls the layer-wise alignment between the sparse and dense hidden representations, and $\beta$ governs the alignment between the output logits. In practice, we recommend setting the weights $\alpha_l$ uniformly across layers and normalizing them such that $\sum_l \alpha_l = \beta = 1$, following practices commonly used in typical knowledge distillation literature [2]. To assess the sensitivity to different supervision strategies, we conducted an ablation study comparing three configurations: (1) using only hidden-state alignment across layers (first, middle, and last layer), (2) using only output logits alignment, and (3) combining both types of supervision. We found that the combined supervision yields the best overall performance, indicating the complementary benefits of guiding both intermediate and final representations. However, layer-wise supervision requires intermediate activations checkpointing in memory. For practical purposes, we adopt logits-only supervision as it achieves a favorable trade-off between effectiveness and resource efficiency.

Table: Ablation on Search Direction for Pruning 50% of Llama-3.1-8B

---

Question 3: " How does the search cost compare quantitatively to existing structured pruning methods such as LLM-Pruner and SliceGPT?"

Response to Question 3: Thank you for the helpful question. As shown in the Table below, LLM-Pruner, FLAP, and SliceGPT are representative post-training sparsity (PTS) methods that feature low optimization cost, making them attractive for fast deployment. However, they typically suffer from limited accuracy. In contrast, our method Týr-the-Pruner without search already outperforms all these PTS baselines in accuracy, while maintaining a similar or even lower optimization time. Furthermore, Týr-the-Pruner achieves the best overall performance across all benchmarks. Although it incurs a search cost, we believe this is a worthwhile trade-off: it produces a significantly more accurate and efficient sparse model without retraining, which is crucial in resource-constrained settings.

Table: Cost and Performance Comparison when Pruning 50% Parameters of Llama-3.1-8B.

---

Question 4: "Could the approach be integrated with other compression methods like quantization or unstructured pruning? If so, have the authors conducted experiments or analyses to evaluate the compatibility or additive effects of such combinations?"

Response to Question 4: Thanks for the insightful question. Our method is indeed compatible with additional compression techniques. As shown in the Table below, we present results of applying quantization and unstructured sparsity on the 50% pruned Llama-3.1-8B model produced by Týr-the-Pruner. These results suggest that our pruned model is highly amenable to further compression: quantization maintains over 99% of the original pruned model's accuracy, while unstructured sparsity at 50% and 2:4 granularity achieves performance levels that are consistent with prior work (e.g., [SparseGPT: 91%@50%, ALPS: 83%@2:4] on 7B models). This compatibility indicates that Týr-the-Pruner can serve as a strong foundation in multi-stage compression pipelines, offering both high accuracy and robustness to further compression.

Table: Integrated with quantization and unstructured pruning for 50% pruned Llama-3.1-8B.

---

[1] Optimal Brain Surgeon: Extensions and performance comparisons. 1993.

[2] Sparse Fine-tuning for Inference Acceleration of Large Language Models. 2023.

[3] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. 2024.

[4] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. 2024.

[5] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. 2023.

[6] ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models. 2024.

---

We sincerely appreciate your thoughtful feedback and support. If there are any points that would benefit from further explanation, we would be glad to elaborate.

Sincerely,

Authors 11393.

Dear Reviewer xbcw:

Thank you for your thoughtful and encouraging feedback. Below, we provide detailed responses to each of your comments and concerns.

---

Weakness 1: "Eq.4 is not very easy to understand. What is the intuition behind Eq.4? What does ~p mean?"

Response to Weakness 1: Equation (4) is derived based on the Taylor expansion for minimizing the pruning error (as shown in Equations. (2-3)). $W_p$ denotes the candidate channel selected for pruning, while $\delta W_{\sim p}$ represents the weight adjustment applied to the remaining unpruned channels ($\sim p$) to compensate for the error introduced by zeroing out $W_p$. For selecting which channel $p$ to prune, we prioritize those with both small gradient magnitude and low curvature (i.e., small entries in the Hessian, c.f., Appendix A.1). Intuitively, a small gradient implies that the weight contributes little to the loss, while a small Hessian entry indicates that the loss is relatively insensitive to perturbations in this weight. Therefore, pruning such a channel is expected to introduce minimal loss increase. For the remaining unpruned weights ($\sim p$), we derive an adjustment $\delta W_{\sim p}$ to compensate for the loss incurred by pruning $W_p$. This is formulated as a constrained optimization problem: minimizing the pruning-induced loss subject to the constraint that $W_p$ is set to zero. Using the method of Lagrange multipliers, we solve for the optimal $\delta W_{\sim p}$ in closed form, as shown in the right-hand expression of Equation. (4) and Appendix A.1. Overall and Intuitively, this formulation ensures that pruning and compensation jointly minimize the estimated error impact.

---

Weakness 2: "How does random error accumulation and abandoned error accumulation do in Lines 149-150?"

Response to Weakness 2: In the context of layer-wise pruning with calibration, error accumulation reflects how pruning decisions in earlier layers affect the inputs to later layers. Specifically, after pruning layer $l$, the input to layer $l+1$ becomes $X_{l+1} = W_l^{\text{pruned}} X_l$, thus propagating pruning-induced errors forward. Since our method explores multiple pruned candidates with different sparsity ratios per layer, we must decide how to accumulate errors. The abandoned strategy discards error accumulation entirely by using the dense model's original activations at every layer, ignoring any pruning-induced errors. In contrast, the random strategy selects one pruned candidate output at random and uses it as the input for the next layer's pruning, simulating a single-path forward pass but without fusion. Both are used as baselines to highlight the effectiveness of our proposed weighted accumulation in Equation (5), which provides a more stable and realistic pruning signal.

---

Weakness 3: "The authors use evolutionary search, which has many operations not optimized for GPU. What is the total search cost compared to sparsity-aware training methods and post-training sparsity methods?"

Weakness 4: "Is this method comparable to or exceeding the sparsity-aware training methods?"

Response to Weakness 3 and 4: To evaluate the trade-offs between efficiency and effectiveness, we conducted a comprehensive comparison targeting 50% sparsity on LLaMA-2-7B using four representative approaches: (1) FLAP, a previous SOTA post-training sparsity (PTS) method; (2) Týr-the-Pruner without search, which applies our local pruning but skips search; (3) NutePrune [1], a training-aware sparsity method based on LoRA and multi-stage knowledge distillation; and (4) Týr-the-Pruner framework. As shown in the table below, PTS methods (FLAP and our non-search variant) complete in under 30 minutes, offering high efficiency but generally lower accuracy compared to training-aware/search methods. Nevertheless, our non-search variant consistently outperforms FLAP, demonstrating the strength of our local pruning strategy. While our full method incurs a higher search cost (~612 minutes), it significantly improves accuracy—surpassing both PTS and training-aware approaches. In particular, it outperforms the training-aware method NutePrune (649 minutes) in most tasks despite requiring no training. Moreover, our method does not rely on large-scale training data (requiring calibration samples only), making it more practical in scenarios with limited supervision. These results highlight that our approach strikes a strong balance between accuracy and efficiency while maintaining minimal data and computational requirements.

Table: Comparison with post-training sparsity and training-aware sparsity methods for pruning 50% of Llama-2-7B’s parameters.

---

Weakness 5: "You use 4 iterations during search. What if you use fewer or more iterations? Will that influence the search results and search time?"

Response to Weakness 5: Thanks for the meaningful question. We conducted an ablation study by varying the number of search iterations from 0 (i.e., no search, equivalent to uniform pruning) to 5, while fixing the final iteration sparsity granularity to 1.5625% in each setting. As shown in the Table below, increasing the number of iterations consistently improves the performance of the searched sparsity pattern, with a smaller number of performance enhancements beyond 3–4 iterations. This suggests that 3–4 iterations are generally sufficient to reach near-optimal sparsity configurations. Each iteration takes approximately 150±10 minutes, and our default setting of 4 iterations offers a good trade-off between search cost and pruning quality. Intermediate results from each iteration in the 4-iteration setting can be found in Table 8 in Appendix A.6.

Table: Ablation Study of Search Iterations when pruning 50% parameters of Llama-3.1-8B.

---

[1] NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models. 2024.

---

We truly appreciate your valuable feedback and continued support. Should any aspect require further clarification, we would be happy to provide additional details.

Sincerely,

Authors 11393.

Dear Reviewer Atmm:

We are grateful for your careful and insightful comments. Please find our detailed responses to your observations and concerns below.

---

Weakness 1: "I believe the models that do not have uniform sparsity will need custom code, which will limit the application of this method in practice. Specially these custom models need to be implemented in serving engines such as vLLM and SGLang, which are used in practice for serving. This is missing from the paper."

Weakness 2: " The authors provide inference speed results for the models from their method, but how does this compare to a uniformly pruned model? A non-uniform model will require invoking several different kernels on the GPU, which will create inference time overheads."

Response to Weakness 1 and 2: We appreciate your concerns regarding the practicality and efficiency of serving non-uniformly pruned LLMs. Our responses are as follows: First, our results demonstrate that non-uniform sparsity consistently outperforms uniform sparsity under the same compression ratio, achieving SOTA accuracy. This performance gap validates a hypothesis that weight importance is not evenly distributed across layers in LLMs. Thus, exploring non-uniform pruning is not merely a technical curiosity but a meaningful direction for understanding and optimizing LLMs.

Second, while we acknowledge that current deployment frameworks like vLLM and SGLang primarily support uniform sparsity out of the box, it is the practical value of non-uniform sparsity that motivates extending these frameworks. A representative example is the recent push to support hybrid and non-uniform architectures in TensorRT-LLM, driven by large-scale Nemotron series models [1, 2]. Our work aligns with this trend and contributes to the ecosystem by identifying practical needs and performance benefits of such models.

Third, we are actively working to extend vLLM to support non-uniform sparsity. This effort involves custom model architecture registration, tailored weight loading, and heterogeneous key-value (KV) cache management. Our support may be adopted by vLLM in the future. A preliminary result from this ongoing work is shown in the Table below.

Table: Inference efficiency of uniform and non-uniform pruned Llama-3.1-8B. Benchmarks were conducted on a single AMD Instinct™ MI350 (288GB) accelerator using vLLM (extended to support non-uniform LLMs, with the temporary unified kv cache space allocation method) for inference, with input and output sequence lengths set to 256.

Finally, as shown in Appendix A.5, we compare the inference efficiency between uniformly and non-uniformly pruned models. The results show that inference efficiency are comparable in most cases. Uniform sparsity can offer slightly higher throughput only when the non-uniform sparsity distribution is highly imbalanced (i.e., large inter-layer sparsity variance), which causes multiple GEMM kernels (Including low-utilization "thin" MatMul) to be dispatched. Nonetheless, both pruned variants achieve substantial speedup over dense baselines, and the accuracy-efficiency tradeoff remains favorable for non-uniform models.

---

Weakness 3: " Are the accuracy numbers reported after distillation using (6)? How about the baselines? The consistency of the baselines need to be discussed."

Response to Weakness 3: We would like to clarify that our method does not involve knowledge distillation training. The formulation in Equation (6) is merely inspired by distillation, serving as a training-free scoring function to search for non-uniform sparsity patterns that better approximate the unpruned model’s outputs. All reported accuracy results are based on these searched architectures. Even without search, our method (Týr-the-Pruner with uniform sparsity) achieves competitive performance, as shown in the Table below (Table 4 and Table 8 also provided relative detailed results). Additionally, to ensure fairness, all baseline methods were re-evaluated using the same calibration dataset as ours.

Table: Týr-the-Pruner Performance with or without Search when pruning 50% parameters of Llama-3.1-8B.

---

Weakness 4: "What does (5) mean? Can you please provide some intuition?"

Response to Weakness 4: Certainly. Equation (5) captures the idea of error accumulation during calibration-included layer-wise pruning. In this setting, pruning is performed layer by layer, where the input to each layer $X_l$ is affected by the pruning of the previous layer. After pruning layer $l$, the output becomes $X_{l+1}=W_l^{\text{pruned}}X_l$, which propagates errors forward. In our method, each weight generates multiple pruned versions with different sparsity levels. Equation (5) fuses the outputs from these variants using a weighted combination, where the weights are normalized sparsity scores—assigning higher weight to lower-sparsity outputs, which tend to be closer to the reliable original dense outputs. This fusion provides a more reliable input for pruning the next layer and better simulates cumulative pruning effects.

---

Weakness 5: "(4) is very close to OBS/OSSCAR/ZipLM. Some discussion is needed here."

Response to Weakness 5: We acknowledge that our method builds upon the same Taylor expansion principle as OBS, which has been widely adopted and validated in recent pruning works such as OSSCAR, ZipLM, SparseGPT, and LLM-Pruner. While large theoretical deviations are unlikely at this stage, our contribution lies in fully leveraging both first-order gradients and second-order Hessian information to guide fine-grained pruning and weight adjustments, pushing the performance boundary further. As shown in Table 4, our ablation study confirms the effectiveness of each component in our local pruning method. Most importantly, our core innovation is the iterative prune-and-search framework, within which the enhanced local pruning serves to build a more reliable supernet for sparsity distribution optimization.

---

[1] Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. 2025.

[2] Llama-Nemotron: Efficient Reasoning Models. 2025.

---

We sincerely thank you again for your thoughtful feedback and support. If there are any particular points that would benefit from further clarification, we would be glad to elaborate.

Sincerely,

Authors 11393.

Dear Reviewer qkjw:

Thanks for your thoughtful and encouraging feedback. We are glad that you found our work valuable. In response to your concerns, we offer the following responses.

---

Weakness 1: "The proposed method requires an additional calibration dataset, however, some baselines do not rely on such a dataset, this makes the comparison unfair."

Response to Weakness 1: We understand the your concern regarding the fairness of comparisons due to the use of a calibration dataset. However, we would like to clarify that (1) all baselines in our main experiments were re-evaluated using the same calibration samples as our method, ensuring fair comparisons. This setup is described in Section 3.1, line 225. (2) For baselines in Table 6 (Appendix A.3), whose official implementations were either released very recently or inaccessible to reproduce, we report results directly from their papers. Since these methods were already carefully tuned, we believe the comparison remains relatively fair. (3) Only a few methods—such as magnitude pruning and PruneNet (a baseline in Table 6)—are not calibration-dependent, but either perform poorly or are designed specifically in a calibration-free manner, making their inclusion reasonable for a comparison.

---

Weakness 2: "The method part could be polished to facilitate comprehension."

Response to Weakness 2: We agree that the method section could be made clearer to improve reader comprehension. In particular, we realize that adding an overview of the overall algorithmic process and providing richer illustrative examples would significantly enhance clarity. We will refine this part accordingly in future revisions.

---

Question: "Given that the matrix $\widehat{\mathbf{W}}$ shares the same dimensionality as $\mathbf{W}$, how can this sparser $\widehat{\mathbf{W}}$ speed up inference? I have this concern because GPUs are optimized for tensor calculation and $\widehat{\mathbf{W}}\mathbf{X}$ and $\mathbf{W}\mathbf{X}$ require the same number of computation operations."

Response to Question: Thank you for the insightful question. We believe the concern about pruning and its impact on actual inference speed is understandable but unnecessary in our case. In our method, the pruning process is initially represented via setting zeros, with weight dimensions kept unchanged during our iterative prune-and-search process (as shown in Equations and Figure 2(b-c)). After the algorithm is finalized, we perform true structural compression by removing the zeroed rows and columns—specifically, reducing the output dimensions of weights of q, k, v, up, and gate, and the input dimensions of weights of o and down. Other modules such as RMSNorm, Embeddings, and the LM-head remain unchanged. The resulting model, illustrated as Figure 2(d), achieves genuine reductions in parameter count and FLOPs, yielding practical speedup without relying on hardware-specific support.

You can view detailed information about the actual acceleration of the post-pruned model in Table 3 (Subsection 3.2) and Figure 4 (Appendix A.5). Or refer to the following simple table to quickly summarize the post-pruning acceleration effect.

Table: Inference efficiency of post-pruned LLMs with Týr-the-Pruner. Benchmarks were conducted on a single AMD Instinct™ MI250 accelerator using PyTorch (HipBlas) for LLM inference, with input and output sequence lengths set to 2048.

---

Please let us know if any specific aspect could be further elaborated. We thank you again for your constructive review and your support of our work.

Sincerely,

Authors 11393.