Reassessing Layer Pruning in LLMs: New Insights and Methods

Hi, thanks for your rebuttal and the new experiments for large model (LLaMA-3 70B) and the ablation for number of sft samples.

The main concern of this paper still remains in the novelty and the contribution. Authors try to compare the reserve pruning with other pruning methods to show the effectiveness of the reserve pruning.

However, the work [1] shows the method of structured pruning globally, and the work [2] also shows the global optimal small dense results. Thus, I am curious about that if the reserve layer pruning can achieve the global optimal pruning results? As there might be some redundancy in the front layers as shown by Figure 3 in work [1] and Figure 3 in work [3]. Could you explain for the global optimum or compare with the works [1] [2]?

Meanwhile, works [1] [2] [4] [5] do not include the fine-tuning (backward) for further performance improvement. While this work adopts Alpaca dataset for the fine-tune, which shows limited generalization (i.e., rely on fine-tune dataset for performance recover). Other works [1] [2] [4] can recover with only WikiText2 dataset and work [5] even does not require calibration. Even the search effort and calibration effort of work [2] remain to the similar as the work [3], thus I think adopt Alpaca for fine-tuning is not that kind of necessary. In my opinion, it is a better way for us to adopt the non-backward method for LLMs to recover model performance as it can be generalized to the extra large models to save the resources. By the way, the works [1] [2] [4] [5] can all generate a smaller model, and are compatible to the further fine-tuning, and I may believe that these works may achieve comparable results as this work after fine-tuning.

[1] Fluctuation-based Adaptive Structured Pruning for Large Language Models

[2] Search for Efficient Large Language Models

[3] LLM-Pruner: On the Structural Pruning of Large Language Models

[4] SliceGPT: Compress Large Language Models by Deleting Rows and Columns

[5] A Simple and Effective Pruning Approach for Large Language Models

I am really respect to the authors' contribution to this paper and the rebuttal, but my concern to this paper's novelty still remains. Therefore, currently, I may retain my score.

I am ready for further discussion.

Most works have done in this paper are kind of ablation study. We appreciate the reviewer's feedback regarding the novelty of our work. While we respectfully disagree with the characterization of our work as "ablation studies," as our research makes two significant contributions to the field:

• Through systematic exploration, we discovered that 'Reverse-order' pruning consistently outperforms more complex pruning methods across diverse datasets and models. This finding is particularly impactful as it establishes a strong, reproducible baseline for LLM layer pruning - demonstrating that simpler approaches can be more effective than complicated pruning strategies. This challenges the common assumption that more sophisticated pruning methods are necessarily better.

• Our second key finding - that fine-tuning only the last few remaining layers and lm_head outperforms popular methods like LoRA - represents a paradigm shift in LLM pruning optimization. This discovery provides practical benefits for efficient model adaptation.

Our work is, to the best of our knowledge, the first to demonstrate these phenomena in LLM pruning, offering significant practical value through both the simple yet effective reverse-order pruning strategy and the empirical insights into the effectiveness of partial fine-tuning. These findings provide clear, actionable guidelines for practitioners working on LLM pruning while challenging common assumptions about the necessity of complex pruning methods and the optimality of LoRA-based approaches.

The simplicity of our proposed methods is actually a strength, as it enables broader adoption and reproducibility while achieving superior results compared to more complex approaches. We believe these contributions advance the field's understanding of LLM pruning and provide practical tools for researchers and practitioners.

Do not include the large models. To address this, we have conducted experiments on the LLaMA-3 70B model. However, due to limited time and computational resources, we are unable to perform fine-tuning experiments on this model and have focused instead on evaluating the performance of the pruned models without fine-tuning. In order to verify the effectiveness of the reverse-order pruning, we compare it with an advanced training-free layer pruning method SLEB [1] (mentioned by Reviewer uw4p). We prune 16 layers with these two methods. As shown in the table, we find that reverse-order pruning and SLEB have similar average performance, which demonstrates the effectiveness of reverse-order pruning. It is important to note that SLEB requires evaluating the importance of each layer, removing the least important layer from the current model, and then iterating through this verification process. This iterative approach of SLBE is time-consuming and takes up a lot of memory for LLaMA-3 70B. In contrast, reverse-order pruning is simple but effective.

The generation speed compared to other models with similar model size included in Table 8. To address this, we have measured the generation speed of other models with similar model size. As mentioned in line 522-524, the latency is tested under the test set of WikiText2 on a single NVIDIA RTX A100GPU. As shown in the table below, our pruned models are significantly faster than existing models with similar model size.

Test the proposed method on the OPT model. Thank you for this valuable suggestion. Following your suggestion, we conduct experiments on OPT-6.7B and Llama2-7B using reverse-order pruning and compare it with [4]. As suggested by the cosine similarity analysis in Figure 2 of [4], OPT-6.7B exhibits high redundancy in shallow layers and Llama2-7B exhibits high redundancy in deep layers. Therefore, we reproduce the method proposed in [4] (Method: Cos) by pruning 2-9th layers for OPT-6.7B, the 22-29th layers for Llama2-7B and fine-tune the pruned models with LoRA on Alpaca-cleaned dataset. For our method, we just prune the last 8 layers in OPT-6.7B and Llama2-7B, freeze the other layers and fine-tune only the last 3 layers and lm_head on Alpaca-cleaned dataset.

From the comparisons on OPT and Llama2-7B, we find pruning the shallow layers of the OPT model is overall better than reverse-order pruning, while the performance improvement is not always consistent as shown on MMLU, CMMLU, and WinoGrande where Cos works slightly worse than reverse-order. We believe the metric in [4] did reveal the redundancy of shallow layers in OPT but the detailed layer index may not always be accurate. For example, [4] suggests pruning the 22-29th layers while experiments in the table below shows pruning the last 8 layers is overall better.

To further demonstrate our effectiveness, we conduct experiments on Llama-3.1-8B-It. As shown in the table, our pruned model achieves an average accuracy of 0.5807, far exceeding the cos method 0.2633. We has cited the paper and added the discussion in our revised version.

[1] Gromov, Andrey, et al. "The unreasonable ineffectiveness of the deeper layers." arXiv preprint arXiv:2403.17887 (2024).

[2] Muralidharan, Saurav, et al. "Compact language models via pruning and knowledge distillation." The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.

[3] Song, Jiwon, et al. "SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks." arXiv preprint arXiv:2402.09025 (2024).

[4] Chen, Xiaodong, Yuxuan Hu, and Jing Zhang. "Compressing large language models by streamlining the unimportant layer." arXiv preprint arXiv:2403.19135

Similar conclusions to Insight #1 and Insight #3.

For Insight #1, authors in [1] find that removing layers beginning at the penultimate layer and proceeding from deep to shallow until the desired number of layers have been removed is effective. In contrast, reverse-order pruning directly cuts off the last few layers without retaining the last layer, which is different from [1]. Besides, we propose to use partial-layer fine-tuning in LLMs rather than the popular LoRA methods, i.e., freezing the other layers and fine-tuning only the last few remaining layers and lm_head. To the best of our knowledge, we are the first one demonstrating the effectiveness of partial-layer fine-tuning in LLM pruning.

For Insight #3：Thank you for your feedback. We acknowledge the insights from [2], and indeed, similar conclusions about iterative pruning provide no added benefit. However, we would like to clarify that the context of our work differs from that of [2]. While [2] demonstrates that iterative pruning is ineffective when using knowledge distillation to retrain the pruned model, our experiments focus on performance recovery of iterative pruning using partial layer fine-tuning and LoRA. As mentioned in Table 5 of our initial submission, Iterative pruning offers no benefit when using LoRA or partial-layer fine-tuning to restore the pruned model performance. It is worth noting that fine-tuning using knowledge distillation is a full-model fine-tuning method, while the LoRA and partial-layer fine-tuning we use are partial-model fine-tuning methods. Therefore, our conclusions are orthogonal to those of [2] and complement each other.

Present the results of the pruned model prior to fine-tuning. Thank you for this insightful suggestion. We conduct additional experiments and compare our method with SLEB [3], which is an advanced training-free layer pruning method (mentioned by Reviewer uw4p) on the Table below. The result demonstrates that our proposed method consistently outperforms SLEB even without fine-tuning, further highlighting its effectiveness. We appreciate your suggestion, as it has helped enhance the comprehensiveness of our evaluation.

Dear Reviewer,

Could you kindly respond and indicate whether authors have addressed your concerns?

Thanks, AC

An analysis of the role and importance of the last layers in LLMs. Thank you for your thoughtful feedback. After pruning, the reduced parameter set may necessitate task-specific representation learning to optimize performance. Fine-tuning the top layers (those closer to the output) and the lm_head proves particularly effective since these layers are most directly associated with task-specific features. Besides, fine-tuning the last layers is a widely recognized approach in transfer learning [1-3], further supporting our claim that last layers are directly associated with task-specific features. LoRA, on the other hand, modifies parameters indirectly and may not achieve the same level of task-specific optimization.

Experimental details. Thank you for your thoughtful feedback. We appreciate your attention to the experimental details and your interest in the reproducibility of our results. We set the batch size and epoch to 64 and 2, respectively. We use the AdamW optimizer. The learning rate is set to 1×10−5 with 100 warming steps. We encourage you to explore and experiment with the default settings in our codes. If there are still questions, we welcome additional questions or requests for clarification. We are committed to supporting reproducibility.

[1] Zhuang, Fuzhen, et al. "A comprehensive survey on transfer learning." Proceedings of the IEEE 109.1 (2020): 43-76.

[2] Jang Y, Lee H, Hwang S J, et al. Learning what and where to transfer[C]//International conference on machine learning. PMLR, 2019: 3030-3039.

[3] Huh, Minyoung, Pulkit Agrawal, and Alexei A. Efros. "What makes ImageNet good for transfer learning?." arXiv preprint arXiv:1608.08614 (2016).

The insight may be seen as incremental. While we agree that fine-tuning specific layers is a well-established practice in model compression and transfer learning on CNNs, we would like to emphasize that our work is, to the best of our knowledge, the first to propose and demonstrate the effectiveness of this approach in the context of pruning large language models (LLMs). As mentioned in Table 2 of our initial submission, our results highlight that fine-tuning only the last few remaining layers and lm_head achieves significant improvements in both performance and efficiency, particularly when compared to the popular LoRA fine-tuning. Notably, our findings highlight a unique advantage of LLM layer pruning, as LoRA and partial-layer fine-tuning exhibit comparable performance during full-model fine-tuning (as shown in Table 3 of our initial submission). This distinction offers new insights and practical contributions to the field of LLM pruning.

We appreciate your recognition of the value of our findings and hope this clarification underscores the novelty of our work.

Comparisons with more recent and advanced layer pruning methods. Thank you for pointing out the importance of comparing with more advanced layer pruning methods such as SLEB and FinerCut. While we recognize the contributions of FinerCut, its implementation is not publicly available, which prevents us from conducting a fair and reproducible comparison. To this end, we conduct experiments with SLEB on Llama-3.1-8B-It and Llama3-8B. For each pruning method, we prune 8 layers. Since SLEB is processed based on the inference-only approach without any finetuning, for a fair comparison, we only prune the model with the last 8 layers without finetuning. As shown in Table below, reverse-order pruning outperforms SLEB, which demonstrates its efficiency.

Besides, to evaluate the effectiveness of partial-layer fine-tuning, we freeze the other layers and fine-tune only the last three remaining layers and lm_head using both Alpaca-cleaned and Dolly datasets. For SLEB, we use the same parameter settings as for partial-layer fine-tuning to perform LoRA fine-tuning on the Alpaca-cleaned dataset. The experimental results demonstrate that fine-tuning specific layers after pruning with the reverse-order method yields significantly better performance compared to fine-tuning with LoRA using the existing SLEB method.

Some of the metrics don’t have a mark to indicate whether the lower or the higher it is： We apologize for the lack of clarity in the original table, which might have confused the interpretation of certain metrics. To address this, we have revised the table to include explicit markers (e.g., ↑ or ↓) to indicate whether a higher or lower value is better for each metric. We sincerely thank the reviewer for pointing this out, as it has helped improve the clarity and presentation of our paper.

Dear Reviewer,

Could you kindly respond and indicate whether authors have addressed your concerns?

Thanks, AC