SlimGPT: Layer-wise Structured Pruning for Large Language Models

We truly appreciate the reviewer for the constructive comments.

W1: Further explanation about Incremental Pruning Ratio strategy.

In Section 5.3.2, we have discussed the impact of different pruning ratio strategies on performance, with detailed results presented in Table 5 (for convenience, Table 5 is reproduced below). Specifically, for the Incremental Pruning Ratio strategy, both logarithmic and linear approaches are employed. Each of these strategies offers distinct benefits: the linear approach achieves better Zero-shot Avg results but incurs a slight loss in PPL. Nevertheless, overall, whether the strategy is linear or logarithmic, the incremental scheme significantly outperforms uniform pruning or decrementing strategies.

Please note that our experiments are conducted under low-resource conditions. After extensive fine-tuning on large-scale data, the differences in performance resulting from various pruning ratio strategies will diminish further (Sheared-LLama[1], even with uniform pruning, mitigated performance impacts through subsequent training on a large data scale). However, under resource-limited conditions, selecting an appropriate pruning ratio strategy can reduce the reliance on subsequent training, thus minimizing performance loss. This is particularly crucial for LLMs, as a complete training cycle demands substantial resources and time.

[1]. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning.

W2: Compare the throughput (eg., tokens per sec.) of pruned models generated by SlimGPT and the competing baselines.

Thank you for your valuable suggestion. About the additional experiments on inference speed, we provide the experimental results and analysis in "global response" at the top. Please refer to that reponse for detailed information.

Q: Under the same total sparsity value, does different increasing schedules affect the model inference speedup?

In general, under the same number of parameters, deeper models (with more layers) tend to have slower inference speeds. On the other hand, models with the same number of layers but different widths primarily experience variations in instantaneous computational load, which impacts speed differently depending on the performance and optimization of the GPU used.

Since our current method typically does not affect the number of layers, as long as there isn't an extreme variation in width distribution, inference speed should not be significantly impacted. This is supported by the experimental results from the previous question.

Dear Reviewer 8hG2:

We sincerely appreciate your valuable and insightful comments. With less than 24 hours remaining in the discussion period, we anticipate further opinion from you.

We would like to further discuss the LLM inference speed. Since most operators are computed in parallel, the acceleration from pruning does not arise from a reduction in computational load but rather from a decrease in parameter access time, which is a significant bottleneck in large model inference. Therefore, the inference time is not linearly related to the parameter number. In our experiments, we find that pruning 50% of the parameters result in an inference speed that is 63% of the original.

Best Regards,
Authors of submission 8375

We truly appreciate the reviewer for the constructive comments.

W1: The impact of the calibration dataset on the pruning performance.

Thank you for the valuable comment. Table 9 displays the pruning results of SlimGPT without fine-tuning (for convenience, Table 9 is reproduced below). When the calibration data is switched from C4 to Alpaca/GPT4-Alpaca, the PPL results indeed degrade (48.26/47.06 vs. 42.06). However, the Zero-shot Avg scores improve significantly (54.44/54.66 vs. 52.70), surpassing the fine-tuned results of all baselines.

This observation highlights a trade-off between PPL and Zero-shot Avg. Due to SlimGPT's inherent parameter compensation mechanism, it is sensitive to the quality of the input data, so there is a distinct difference in impact between pre-trained data and instruction-following data. Additionally, our experiments show that random open-source pre-trained data can achieve SOTA results. Therefore, we believe that using higher-quality data tailored to specific domains provides SlimGPT with greater potential for improvement compared to other methods.

W2: The design of the layer-wise sparsity is empirical.

The core concept of SlimGPT is derived from the OBS framework, which addresses global model pruning by breaking it down into layer-wise subproblems. Each layer is optimized sequentially from shallow to deep. Previous studies, such as OBC[1] and GPTQ[2], have demonstrated that this method yields excellent results across various domains. And our primary objective is to apply this framework to the structured pruning of LLMs.

We would like to discuss the viability of the layer-wise greedy pipeline. Due to the unidirectional influence between layers, the current layer is impacted solely by the preceding layer and remains unaffected by subsequent layers. If the pruning process is not executed sequentially, the local optimality at each step would be compromised. We believe this would complicate the task and significantly increase the computational time required.

[1]. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. [2]. GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers.

W3: The inference speed should be provided.

Thank you for your valuable suggestion. About the additional experiments on inference speed, we provide the experimental results and analysis in "global response" at the top. Please refer to that reponse for detailed information.

Q1: Further explanation on Figure 1.

As a structured pruning scheme, SlimGPT prunes attention blocks with the smallest pruning granularity at the head level. Consequently, heads are treated as indivisible units. We evaluate each head by summing the errors of all columns within it. For those interested in the specifics, our detailed algorithmic process is outlined in Algorithm 1.

Q2: Can you provide some insights about the reason why the pruning results with 2048 samples and 2048 sequence length start to decrease in zero-shot average metric?

Thank you for the comment. The Zero-shot Avg is essentially a mean value, which can be easily influenced by sub-tasks with significant fluctuations. To facilitate analysis, we provide detailed results for the commonsense reasoning tasks below.

As the sample length increases from 64 to 1024, there is a noticeable improvement in the metrics across various tasks. However, when the length is further increased to 2048, the rate of improvement slows down, and some metrics even decline, particularly for WinoGrande (59.27 vs. 57.3). We believe that since most commonsense reasoning tasks involve short text data, the impact of SlimGPT on these tasks may diminish when the sequence length of the calibration set exceeds a certain threshold. In fact, it could even weaken the model's understanding of short texts.

When the sample size increases, the changes across the various subtasks are less consistent and exhibit smaller magnitudes. As shown in Figure 3 of the paper, the fluctuations in Zero-shot Avg in subplot (a) are significantly smaller than those in subplot (b), and there are two instances of decline. Both of these declines result from fluctuations in the BoolQ dataset (refer to the table above). Thus, we hypothesize that the BoolQ dataset is particularly sensitive to random sampling in the calibration set, which may result in the drop in Zero-shot Avg.

Dear Reviewer QvFZ:

As the discussion period comes to a close, we sincerely look forward to your feedback on our rebuttal. Your further opinions would be essential for us to improve our work.

Regarding your concern that the calibration set is more important than the pruning method, we include additional experiments with LLM-Pruner (our baseline). We perform pruning without fine-tuning at the same pruning ratio, yielding an evaluation result of PPL=136.19. In contrast, the worst result for SlimGPT is PPL=48.26. Therefore, we believe that while the calibration set may have a slight influence on the bias of model pruning, it does not fundamentally affect the pruning results.

Best Regards,
Authors of submission 8375

We truly appreciate the reviewer for the constructive comments. Due to text limitations, we try to answer your questions concisely and convincingly.

W1: Unaligned evaluation.

We acknowledge your concerns. Achieving a fully aligned experimental setup is challenging since pruning tasks differ from tasks like model optimization, and various pruning methods have unique principles and data requirements. For instance, LLM-Pruner has two stages: pruning and fine-tuning, while Compresso and LoRAPrune involve only one stage of sparse training. Our method, SlimGPT, also follows a two-stage process, aligning it with LLM-Pruner. Given the differing principles of Compresso and LoRAPrune, we make a compromise in terms of aligning the experimental environment. However, we believe that this does not affect the conclusions of our paper. In fact, we achieve better pruning results using fewer data resources or fewer iterations.

W2: Overly emphasis on the Llama family.

Thank you for your valuable comments. In our model selection, we reference previous works and the influence of LLama and Vicuna models but overlooked Vicuna's derivation from LLama. Due to time constraints, we choose Baichuan-7b, a prominent model in the Chinese community, for our experiments. This model includes a comprehensive MMLU and C-Eval evaluation script and is supported by LLM-Pruner, aiding our quick verification process.

Using the same setup as the paper, we prune Baichuan-7b by 20% with LLM-Pruner and SlimGPT. SlimGPT achieves slightly better PPL results (19.73 vs. 19.85) and significantly outperforms LLM-Pruner in all commonsense reasoning tasks (58.8 vs. 56.38).

W3: Only on zero-shot commonsense reasoning tasks.

Thank you for your suggestions. We evaluated our model based on the official LLama report and previous research, focusing on language modeling performance (PPL) and zero-shot commonsense reasoning. While we believe these tasks are convincing, we recognize they may be insufficient for a complete assessment. To improve our evaluation, we perform 5-shot tests on Baichuan-7b using MMLU and cross-lingual task C-Eval, as seen in the table below.

The 5-shot evaluation results for MMLU show SlimGPT outperforming the baseline (35.4 vs. 24.3), and it also excels in the cross-lingual C-Eval dataset (28.7 vs. 22.4), despite reduced performance after finetuning on Alpaca.

Regarding your suggestion for experiments on long-context performance, we recognize their importance but, due to time constraints, we won’t be able to conduct them at this moment. We plan to include them in future work for a more comprehensive study.

W4: Incomplete efficiency report.

Thank you for your suggestion. We include the experimental results and analysis on inference speed and pruning efficiency in the "global response" section, as other reviewers have similar questions. Please refer to that for detailed information.

W5: Detailed introduction of the compared baselines.

Due to space constraints, we have omitted the baseline introduction in the final paper version. We apologize for any inconvenience and will include it in the updated version.

Q1: Further explanation on Figure 2.

Figure 2 shows the output errors of various pruned models relative to the unpruned model, which has an error of zero and is not displayed (its PPL is 12.63). In our pruning approach, we utilize SlimGPT with the Alpaca dataset, pruning only the first layer to minimize output errors.

To assess performance recovery after finetuning, we present the PPL results below, where all models are finetuned with LoRA for one epoch using Alpaca. We observe that the PPL does not improve after finetuning, even for the unpruned model. We believe this may be due to the distinct distribution bwtween the instruction-following dataset Alpaca and test dataset Wikitext2, potentially causing overfitting of the unpruned weights and poorer performance on the test data, which requires further validation.

Q2: Is a pruned model superior to a smaller pretrained model?

This topic deserves discussion. Under low-resource conditions, pruned models after finetuning generally perform worse than smaller pre-trained ones, as shown in our works and previous research(LLM-Pruner). However, with full training, pruned models can outperform their smaller versions, like in Sheared-LLama[1]. Thus, pruning may serve as a high-benchmark initialization method to lessen the need for extensive training.

Our aim is to focus specifically on the task of LLM pruning itself. Under constrained resource conditions, such as when a more compact version of the model (e.g., less than 1B parameters) is required for edge-side deployment, LLM pruning and compression provide a cost-effective solution.

[1]. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning.

I thank the authors for the detailed rebuttal. The new results — especially on the efficiency front — look decent and thus warrant a slight bump to 5. However, to fully convince me, I'd like to see SlimGPT applied on truly challenging tasks like GSM8k, as well as long context tasks like LongBench and Needle-in-a-Haystack (on llama3 and mistral v0.2). I am particularly interested in the long context front due to the known drawbacks of ShortGPT.

With the unaligned nature discussed in W1, it looks like the proposed method is mostly compared to LLMPruner. While I recognize that LLMPruner is an established method, I wonder how SlimGPT would perform against some of the more advanced methods, like APT [2].

p.s. While I appreciate the addition of Baichuan, I would still like to see the MMLU report on a more mainstream model, like llama2-7b, for better cross-referencing needs. May the authors supply that?

[2] APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference

I'll take a closer look at the rest soon, but may the author please address the question as titled? Just want to post earlier so that you can have a chance to reply. I'd also like to see APT on GSM8k under a comparable setting.

Thank you for being resourceful and adding many requested experiments during the rebuttal time. The added result confirms my intuition: that cheap, non-ShreadLlama-like LLM pruning techniques do not perform well under rigorous evaluation. Your added results on GSM8k and LongBench confirm that, as there are visible drops with just 20% pruned. Note that we usually don't observe such a performance drop with techniques like weight-only quantization at a much more aggressive rate; even with vanilla group-wise quantization with no finetuning.

That being said, I recognize that pruning LLMs is much harder than quantifying LLMs. There surely are some benefits unique to the pruning way, and overall, pruning is without a doubt a school of efficiency worth developing; especially knowing its gap with quantization. The proposed method is better or at least on par with the established/recent baselines, so I recommend an acceptance with score 6 & confidence 5. But I urge the authors to:

• Tone down the claim a bit, e.g., the "98% performance" claim in your abstract is slightly misleading. Most common-sense reasoning tasks are easy and do not represent LLM's true capability, so it is almost an overstatement based on cherry-picked results.
• Highlight the results that are not perfect (MMLU, GSM8k, LongBench, etc.) so that future works will have a clear direction for improvement, instead of always muddling those easy tasks.
• Add a proper section to discuss the pros/cons of pruning compared to other efficiency techniques (e.g., quantization) and their unique challenges.