Streamlining Redundant Layers to Compress Large Language Models

Thank you for your thorough review and the constructive feedback on our manuscript. We have carefully reviewed and considered each of the questions and suggestions you have raised. Below, we provide detailed responses to your comments. In the content below, we use W to represent Weakness and Q to represent Question.  

W1: Comparison of the acceleration effect of layer replacement with pure layer pruning.

We are sorry for the misunderstanding. In our experiments, under the same sparsity constraint, when using a Transformer layer as the lightweight network, the pruned model's structure remains identical to that of pure layer pruning. As a result, the acceleration effect of layer replacement is equivalent to that of pure layer pruning. While when using an MLP as the lightweight network, our pruned model includes a layer that omits the attention architecture, resulting in a slightly better acceleration effect compared to pure layer pruning.

We also report the inference speed (in output tokens per second) in Table 11 of the appendix in the revised paper (We have also put this table at the bottom of comments.) . The results indicate that our approach (layer) — replacing with Transformer layer— achieves the same speed (25.68 tokens/s) as our approach (None) — pure layer pruning. Additionally, our approach (FFN) — replacing with MLP — is slightly faster, at 25.88 tokens/s, compared to pure layer pruning.

W2: Regarding how many layers are removed at once in Equation 3.

Thank you for your question. The number of layers removed is determined based on the specified sparsity. For instance, in our experiments, we set the sparsity to about 25%, meaning we prune layers equivalent to 25% of the total parameters at once. For example, Llama2-7B has 32 layers, so pruning 25% of the model's parameters is roughly equivalent to removing 8 layers.

W3: one-shot pruning or iterative pruning

We are sorry for the misunderstanding. All of our experiments use the one-shot pruning approach that you mentioned. Iterative pruning—removing one layer at a time and replacing it with a lightweight network—would not significantly reduce the overall number of parameters. This is because a lightweight network, such as an MLP, contains roughly two-thirds the parameters of a single Transformer layer. Therefore, replacing each layer individually with a separate lightweight network, rather than using one lightweight network to replace multiple layers, would not lead to a significant reduction in parameters. On Llama2-7B, to achieve approximately 25% sparsity by using different lightweight networks to replace each layer, we would need to train about 24 lightweight networks to replace 24 Transformer layers. This results in a significantly high training cost.

W4: Comparison between MSE loss and the original language model loss

Thank you for raising the comment! In terms of computational resource usage, training with MSE loss only requires storing the lightweight network in memory, without needing to load the entire LLM. In contrast, training with the original language model loss requires loading the full LLM into memory. By using MSE loss, we minimize the memory requirements during training, making LLM-Streamline feasible in memory-constrained environments.

W5: Experiments on higher sparsity level

Thank you for your suggestion. We have conducted experiments at higher sparsity levels and included the results in Appendix E.7 of the revised paper. These results demonstrate that LLM-Streamline remains significantly stronger than the baseline method, even at a high pruning ratio of approximately 50%.

Table 1: The inference speed of models pruned using different methods.

Dear Reviewer MF5H,

Thank you for taking the time to review our paper. We sincerely appreciate your valuable feedback and have provided detailed responses and additional experiments to address your concerns. Kindly let us know if you have any further questions. Your insights are invaluable to us and will undoubtedly help us further improve the quality of our work.

Once again, thank you for your time and thoughtful consideration.

Best

The Authors

Thank you very much for your thorough review and valuable feedback on our manuscript. We have carefully reviewed and considered each of the questions and suggestions you have raised. Below, we provide detailed responses to your comments. In the content below, we use W to represent Weakness and Q to represent Question.  

W1 and Q1: Comparison of Cosine Similarity with Other Metrics

Thanks for your question. In lines 159-176 of our paper, we detail our rationale for selecting cosine similarity over other metrics. Since most modern LLMs utilize a post-norm architecture, this results in the norm of the hidden states increasing with the depth of the layers. This leads to a bias where deeper layers tend to exhibit higher dot product similarity, while earlier layers show smaller Euclidean distances.  

Regarding perplexity, we find it to be a highly sensitive metric. As shown in our experiments (see lines 756-783 in the appendix), using perplexity as a pruning metric leads to the selection of different pruning layers depending on the pre-training dataset. Moreover, we compare the performance of models pruned using perplexity with those pruned using cosine similarity. The results suggest that models pruned using cosine similarity outperform those pruned with perplexity.    

W2 and Q3: Comparison of LLM-Streamline and LoRA

Thanks for your suggestion. In lines 199-215 of our paper, we highlight the advantages of LLM-Streamline over LoRA. Specifically, LLM-Streamline offers lower memory consumption and faster training speeds. Moreover, the lightweight network used in LLM-Streamline is trained through distillation, resulting in enhanced performance compared to LoRA. Furthermore, in lines 473-485, we present a detailed comparison between LLM-Streamline and LoRA, with experimental results showing that LLM-Streamline consistently outperforms LoRA in both performance and resource efficiency.  

W3: A simplified explanation of stability

Thank you for your suggestion. We can easily explain our motivation for introducing the concept of stability. After pruning, a model may correctly answer questions that it previously answered incorrectly in classification tasks based on perplexity (PPL), which can even lead to an increase in accuracy for certain tasks. However, this does not necessarily indicate that the model’s overall capability has improved after pruning. Therefore, we propose stability as a metric, which focuses on the consistency of the model’s responses before and after pruning. We think it can provides a more accurate reflection of the pruned model’s performance than accuracy alone.  

Q2: Examples that illustrate the advantage of stability over accuracy

Thank you for your valuable suggestion. For example, in PIQA, there is a question like this:

The following makes sense:

A. Question: How to make tissue paper window decorations? Answer: Find tissue paper and fold it in half. Take scissors and cut out pieces of the paper in the middle. When you are done, tape it to your window.

B. Question: How to make tissue paper window decorations? Answer: Find tissue paper and fold it in half. Take scissors and tear out pieces of the paper in the middle. When you are done, tape it to your window.

Llama2-7B has a very similar PPL for both options, indicating great uncertainty in this question and choosing the wrong answer B. However, the pruned Llama2-7B still has a very similar PPL for both options but manages to guess the correct answer A. This situation reflects the limitations of accuracy, whereas stability can well identify such cases, thus more accurately reflecting the performance of the pruned model.

Thank you very much for your thorough review and valuable feedback on our manuscript. We have carefully reviewed and considered each of the questions and suggestions you have raised. Below, we provide detailed responses to your comments. In the content below, we use W to represent Weakness and Q to represent Question.

W1 and W2: Experiments on higher sparsity level, more diverse architectures, and larger models

Thank you for your suggestion. We will conduct experiments at higher sparsity levels, with more diverse architectures, and on larger models, and we will strive to update the paper with these results before the end of the rebuttal period.

W3: Result of real hardware performance

Thank you for your feedback. We have included the results for inference speed in the Appendix of the revised paper(See Table 11). We have also put this table at the bottom of comments.

Table 1: The inference speed of models pruned using different methods.

Dear Reviewer M4rG,

We have conducted experiments with Llama2-7B at a 50% pruning rate, as well as additional experiments on Llama3.1-8B, Llama3.1-70B, and Mixtral-8x7B. The results are detailed in Appendices E.4, E.5, and E.7 of revised paper. As demonstrated, our proposed method, LLM-Streamline, consistently outperforms the baseline approaches.

Best

The Authors

Thank you very much for your thorough review and valuable feedback on our manuscript.  

In response to your question, “Why didn’t you compare with some knowledge distillation methods for LLM?”, we appreciate your insight. While both distillation and pruning are model compression techniques, they serve different purposes. Distillation is primarily used to improve the training of smaller models, whereas pruning focuses on directly reducing the size of a large model. Although some pruning methods, such as the LLM-Streamline approach we proposed, involve the training process and can incorporate distillation loss during this phase, the two methods can complement each other. Therefore, papers introducing distillation methods, like DistiLLM, typically do not compare with pruning techniques, and vice versa.