Computation and Memory-Efficient Model Compression with Gradient Reweighting

W1: Explanation for "LLMs pose a unique challenge for pruning".

As we mentioned in the main text, unlike conventional DNNs, LLMs face unique challenges in pruning, primarily due to the asymmetry in resources (data volume and memory) between training and pruning. Conventional DNNs, e.g., ResNet for image classification, can typically be pruned under resource conditions similar to those used during training, such as comparable datasets and memory (e.g., ImageNet dataset and 4 GeForce RTX 4090), which is always affordable for the downstream practioners. In contrast, training LLMs often relies on extremely large-scale computational resources and massive data. For example, LLaMA3-405B was trained with up to 16K H100 GPUs and approximately 15T of training tokens [r1], which far exceeds what is accessible during the pruning phase. This characteristic of LLMs therefore presents unique challenges for pruning methods, distinct from those encountered in conventional DNNs.

W2: Theoretical parts in Section 4 are dense and hard to follow.

We will move some of the theoretical parts to the appendix in the revision.

W3: Comparison with optimization-based baselines is limited.

Thanks for your suggestion. We additionally include experiments based on the optimization-based pruning method Compresso, NutePrune, LoRAShear [r2], MaskPrune [r3] with different models and prune rates. More comparison will be included in the revised version if accepted. For a fair comparison under equal training time, we extend the training epochs of our method to fully exploit its effienciy. We additionally report the time and memory overhead incurred by each method during the pruning process. The results are summarized below.

Table R1. Comparison with various optimization-based methods on Llama2-7B. To ensure a consistent training time budget, we extend the training epoch accordingly. Meanwhile, we also record the memory overhead for reference. The fine-tuned version is trained on the Alpaca dataset.

Table R2. Comparison with various optimization-based methods on Llama-7B. Zero Shot task includes BoolQ, PIQA, HellaSwag, Winogrande, ARC-e, ARC-c and OBQA.

W4: Typos.

We will correct these typos in the revision.

L5: Missing discussion of limitations and broader impacts.

Thanks for the reminder. In fact, we discussed the limitations and broader impacts in Appendices F and G. We will move them into main text in the revised version if accepted.

Reference

[r1] Dubey A, Jauhri A, Pandey A, et al. The llama 3 herd of models. arXiv e-prints, 2024: arXiv: 2407.21783.

[r2] Chen T, Ding T, Yadav B, et al. Lorashear: Efficient large language model structured pruning and knowledge recovery. ArXiv 2023.

[r3] Qin J, Tan J, Zhang K, et al. MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures. ArXiv 2025.

W1: Results beyond 50% sparsity.

We provide experimental results under 55% and 60% prune rates, as shown in the Table R1.

Table R1. Performances with higher prune rates on Llama2-7B.

W2: The experimental performance gap in SliceGPT and LLM-Pruner.

The reason of the performance difference is we use different experimental setup for fair comparison.

Before presenting our detailed setting, we'd like to clarify the following three factors:

1. SliceGPT uses WikiText2 as both the calibration and evaluation dataset, which makes it difficult to fully evaluate the generalization ability of the pruned model. Such experimental setup is also not wildely adopted in the literature.
2. Regarding the performance difference on LLM-Pruner, there may be a misunderstanding: the original paper used the LLaMA-1 with a evaluation context length of 128, whereas our experiments are based on the LLaMA-2 and LLaMA-3 with a context length of 2048.
3. Pruning methods such as SliceGPT and LLM-Pruner can have notable performance variation over different calibration datastes and input sequence lengths. As shown in Table R2 below, it achieves an average score of 51.50 when using WikiText2 as the validation set, compared to 57.93 when using Alpaca. Figure 7 of SliceGPT further illustrates that when varying the length of the calibration dataset, the WikiText2 PPL of OPT-6.7B fluctuates within a range of 12.0 to nearly 20.0.

Table R2. The results extracted from Tables 7–10 in the original SliceGPT paper demonstrate that the performance of LLaMA2-7B at a 30% prune rate varies significantly depending on the calibration data used. "-" means the result is not available in the original paper.

Our experimental settings. To ensure fair comparison across all baselines in our experiments, we standardize the calibration dataset to C4 and use a sequence length of 128. For evaluation, we use WikiText2 and Ptb as test sets with a length of 2048, and we do not apply fine-tuning.

Results under the same setting with SliceGPT. Furthermore, we evaluate our method under the same experimental setting as SliceGPT, where WikiText2 is used as both the training and evaluation set. Under this experimental setup, our method also surpasses SliceGPT on multiple models. The results are shown in the following table.

Table R3. Performance of our method, with training and evaluation conducted on Wikitext2(↓).

W3-1: Comparison with optimization-based baselines is limited.

Thanks for your suggestion. We additionally include experiments based on the optimization-based pruning method Compresso, NutePrune, LoRAShear [r1], MaskPrune [r2] with different models and prune rates. More comparison will be included in the revised version if accepted. For a fair comparison under equal training time, we extend the training epochs of our method to fully exploit its effienciy. We additionally report the time and memory overhead incurred by each method during the pruning process. The results are summarized below.

Table R4. Comparison with various optimization-based methods on Llama2-7B. To ensure a consistent training time budget, we extend the training epoch accordingly. Meanwhile, we also record the memory overhead for reference. The fine-tuned version is trained on the Alpaca dataset.

Table R5. Comparison with various optimization-based methods on Llama-7B. Zero Shot task includes BoolQ, PIQA, HellaSwag, Winogrande, ARC-e, ARC-c and OBQA.

W3-2: Performance differences on NutePrune. Does this difference come from the lack of fine-tuning?

Yes. To ensure a fair comparison, we did not perform additional fine-tuning when reproducing the experiments of NutePrune, which may lead to discrepancies from the results reported in the original paper.

W4: Explanation for the difference.

For the differences in certain experimental results, please refer to the responses in W2 and W3-2.

W5: Why the authors only compare the results without fine-tuning?

Thanks for your suggestions. We provide the fine-tuning results in Table R4 and Table R5. For fair comparison, we adopt the same setup as LLM-Pruner: LoRA fine-tuning is performed on the pruned model using the Alpaca dataset for 2 epochs, with a rank of 8 and a learning rate of 1e-4.

W6: Statistical information.

Thank you for your valuable suggestion. Actually, we observed that the experimental results exhibited low variance in the initial training stage, so we fixed a random seed for all subsequent experiments. To illustrate this, we conducted statistical experiments using five different random seeds on both Llama2-7B and Llama3-8B. The results, shown in the table below, indicate that the standard deviation is relatively small compared to the performance gains. Complete results with statistical information will be included in the revised version if accepted.

Table R6. Statistical information over 5 random seeds.

Reference

[r1] Chen T, Ding T, Yadav B, et al. Lorashear: Efficient large language model structured pruning and knowledge recovery. ArXiv 2023.

[r2] Qin J, Tan J, Zhang K, et al. MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures. ArXiv 2025.

I thank the authors for their detailed response. The explanations and additional experiments are very helpful and have addressed my concerns. I'm very happy to increase the rating.

W1: Further research is needed to determine if the trade-off between LLM complexity and efficiency justifies this approach to LLM pruning.

We'd like to clarify as follows.

Reasons on Unexplored Pruning for LLMs. We observe that LLMs often exhibit performance degradation after pruning, especially under high sparsity structural pruning. We argue this phenomenon may be attributed to two main factors:

1. A mismatch between pruning and training resources. The data scale and computing resources used for pruning are much lower than those used for training, which hinders its ability to achieve the desired effectiveness.
2. Potentially lower redundancy in LLMs compared to vision models. One possible explanation is that language data contains less redundancy than image data, in which local pixel regions typically exhibit high similarity. However, this remains unclear under the current limited pruning resources.

Based on the above background, we emphasis the key contributions of our work as follows:

1. The memory cost during pruning is significantly reduced by our method, making it more feasible to prune larger models with the same memory resource. This is important for the downstream practioners of LLMs, where the resource is always limited. As shown in Table R1, our method incurs very low memory overhead.

2. Our method has higher effiency. We emphasize that improving pruning performance is closely tied to the efficiency of the algorithms, especially for the large scaled networks like LLMs. A more efficient pruning algorithm allows for more iterative updates, enabling more sufficient exploration and refinement on sparse structures. As shown in Table R1, extending the training time to match that of the baselines leads to more significant performance superority.

3. Our pruning method achieves SOTA performance compared to the baselines.

Table R1. Comparison of pruning performance, training time, and memory overhead under 50% prune rate of Llama2-7B. To ensure fair experimentation, we extended the training schedule of our method to match comparable time budgets.

Q1: What are the potential resource requirements for super LLMs to demonstrate the limitations of pruning?

For medium-sized networks, i.e., ResNet for image classification, existing pruning algorithms often require resource investments comparable to full training in order to achieve satisfactory pruning results, like 4 GeForce RTX 4090. By analogy, for LLMs with more complex parameter structures and larger parameter scales, it is reasonable to hypothesize that approaching the limit of pruning performance similarly demands computational resources and data volumes on par with pre-training.

Q2: Theoretically, at what scale does network growth imply sufficient redundancy to make pruning worthwhile?

Most existing neural network inherently contain a certain degree of redundancy, as they are manually designed and cannot guarantee that each parameter is optimally utilized to represent knowledge. Consequently, pruning is essential for improving efficiency in applications where extreme time efficiency is required.

Q3: Does the current distributed sparse training framework necessitate full-synchronization for optimization?

Our framework can be integrated with both synchronized and asynchronized optimization flexibly. It is currently designed as a synchronized optimization process, since synchronized optimization is widely used in distribued training systems.

To support asynchronous optimization, only Step 8 in Algorithm 2 needs to be modified. Instead of waiting for all workers to return their parameter updates $\Delta W_{m(i)}$ before updating the global model, the server can update the global weights using each received gradient individually as follows:

For each incoming $\Delta W_{m(i)}$ from worker i, update the global weights as:

$\qquad\qquad\qquad\qquad\qquad\qquad W \leftarrow W + \eta_{3} \Delta W_{m(i)},$

where $\eta_{3}$ is a small learning rate that controls the pace of asynchronous updates.

This minor modification would allow the framework to run fully asynchronously without requiring a complete redesign, while still benefiting from the lightweight communication scheme enabled by the sparse structure and reweighting strategy.

Q4: If asynchronous training is implemented, what adjustments are required? Would a separate framework be necessary to accommodate asynchronous pruning?

The asynchronous version can be implemented following the answer in Q3, without the need for a separate training framework. We provide experimental results of asynchronous sparse training on ResNet-50, as shown in Table R2.

Table R2. Comparison of asynchronous sparse training results

W1: Repeated emphasis on three main contributions.

We will revise the manuscript to improve clarity and make it easier for readers to follow.

W2: Lack of comparison with optimization-based methods.

Thanks for your suggestion. We additionally include experiments based on the optimization-based pruning method Compresso, NutePrune, LoRAShear [r1], MaskPrune [r2] with different models and prune rates. More comparison will be included in the revised version if accepted. For a fair comparison under equal training time, we extend the training epochs of our method to fully exploit its effienciy. We additionally report the time and memory overhead incurred by each method during the pruning process. The results are summarized in Table R1 and Table R2.

Table R1. Comparison with various optimization-based methods on Llama2-7B. To ensure a consistent training time budget, we extend the training epoch accordingly. Meanwhile, we also record the memory overhead for reference. The fine-tuned version is trained on the Alpaca dataset.

Table R2. Comparison with various optimization-based methods on Llama-7B. Zero Shot task includes BoolQ, PIQA, HellaSwag, Winogrande, ARC-e, ARC-c and OBQA.

W3: The variance reduction in Table 5 seems to degrade the performance.

There may be some misunderstanding. As shown in Table 5, removing the components of our variance reduction technique (either the clipping operation or the preconditioning matrix H) leads to a performance degrade. "Ours" in Table 5 refers to our pruning method equipped with the full variance reduction technique. It is important to note that for WikiText2 and Ptb, the evaluation metric is perplexity, where lower values indicate better performance, whereas for PIQA, HellaSwag, ARC-e, and ARC-c, the metric is accuracy, where higher values are better. Taken together, these results demonstrate that the variance reduction strategies contribute positively to performance and remain an integral part of our method.

Q1: Is the $m$ in Equation 1 either 0 or 1? Can we prune part of the each sub-module?

First, yes, in Equation 1, $m$ is either 0 or 1, representing the retention state of a sub-module. Then, we can prune part of each sub-module by setting finer-grained mask. In existing works (e.g., Sheared LLaMA [r3] and FLAP [r4]), structural pruning is typically applied at the level of attention heads or the intermediate dimensions of FFNs, as such sparsity patterns can be efficiently implemented on various hardware platforms to achieve speedup. Therefore, our method adopts these default settings. However, thanks to the flexibility of our masks, our approach can also be extended to support finer-grained pruning. For instance, within each attention head, we can assign individual masks to the intermediate dimensions corresponding to Q, K, and V, enabling more fine-grained pruning strategies.

Reference

[r1] Chen T, Ding T, Yadav B, et al. Lorashear: Efficient large language model structured pruning and knowledge recovery. ArXiv 2023.

[r2] Qin J, Tan J, Zhang K, et al. MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures. ArXiv 2025.

[r3] Xia M, Gao T, Zeng Z, et al. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. ICLR 2024.

[r4] An Y, Zhao X, Yu T, et al. Fluctuation-based adaptive structured pruning for large language models. AAAI 2024.