SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters

Thank you for your recognition of our paper and your constructive feedback. We have responded to your concerns and will revise our paper based on the discussions. We would also appreciate it if you could let us know if our response addresses your concerns.

Q1: Despite the improvement in speed, the performance loss in downstream tasks is still significant, raising concerns regarding the practical applicability of the SHARP method in real-world scenarios where maintaining high levels of accuracy is paramount.

A1: Please refer to ‘All-A3’ in the ‘Reply to all the reviewers’ part. Besides, in ‘All-A2’ we also show that this may be a general problem for all the structural pruning methods, while SHARP has achieved better recovery performance over other structural pruning baselines consistently.

Q2: Ambiguity in Communication Definition: The article does not clearly define what is meant by "communication" in the context of parameter sharing. It is essential to clarify whether it refers to internal model communication, information flow between layers, or external communications between distributed components.

A2: Thanks for pointing out the definition that may be confusing. In our paper, ‘communication’ mainly refers to loading the model weights from low-speed, high-capacity memory to high-speed, low-capacity computation caches. For example, on mobile devices, ‘communication time’ denotes the loading time of model weights from memory to RAM/DRAM; while on server, it can denote the loading time from CPU memory to CUDA device (like ‘.to(device)’ process). We will add this in the revised paper.

Q3: The authors failed to include comparisons with alternate model compression techniques, such as LLM pruning, despite they also attempt to drop the LLM parameters while retaining performance. Such a comparison could have provided a more comprehensive understanding of how this new method stacks up against traditional approaches in terms of parameter reduction, tuning overhead, and performance impact.

A3: Thanks for your constructive suggestions, we try three more advanced structural pruning methods: LayerPruning, LayerPruning-Adapted, and LLM-Pruner, and show that SHARP performs consistently better than them. Please refer to ‘All-A2’ in the ‘Reply to all reviewers’ part for the details.

Thank you for your recognition of our paper together with your valuable comments and suggestions. We will revise our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.

Q1: ...in Table 6, while SHARP improves accuracy over Direct Sharing and even surpasses the original on the CommonsenseQA task, its accuracy significantly degrades compared to the original across most tasks, raising doubts about its overall utility.

A1: Please refer to ‘All-A3’ in ‘Reply to all reviewers’ for illustration. In short, the downstream degradation is a general problem for all the structural pruning methods (like LayerPruning[1], LLM-Pruner[2], and MobileLLM[3]), and SHARP has shown the best recovery performance on all recovery tasks in Table A2 from the ‘Reply to all reviewers’ part.

Q2: Is the accuracy of a large model reduced by SHARP higher than that of a smaller model with an equivalent number of parameters? For instance, if the LLaMA2-13B model is reduced by 7B parameters using SHARP, does this new model outperform the existing LLaMA2-7B in accuracy?

A2: First, the main issue of the performance drops is discussed in A1 and ‘All-A3’ in ‘Reply to all reviewers’. We are sorry that due to the restriction of computation resources and time, we are unable to verify if applying SHARP on the LLaMA2-13B can outperform the existing LLaMA2-7B model on our current GPUs. However, we still have the following reasons to support the advantage of SHARP:

1. The downstream degradation is a general problem for all the structural pruning methods, and compared with advanced structural pruning methods[1,2], we have consistently better recovery performance on all tasks as shown in Table A2 in the ‘Reply to all reviewers’ part. Our analysis of performance drops (Figure 4 in the main paper) can also be helpful for enlightening future work to solve this general problem of performance drops.

2. As illustrated in the ‘OUR POSITION’ part in the ‘Reply to all reviewers’, SHARP is mainly designed for accelerating model inference by reducing the memory load overhead and is especially helpful on mobile devices which always have small RAM to load the whole model. Therefore, comparison on smaller models is more meaningful for SHARP, and in A6 we also show that SHARP is useful for the recovery of smaller models like LLaMA3.2-3B.

3. As shown in ‘All-A1’ in the ‘Reply to all reviewers’ part, we see quantization method is orthogonal to and compatible with SHARP, therefore, they can combined together for model compression for inference on mobile devices.

Q3: …This paper’s approach removes certain layers from a pre-trained model, introducing the risk that training may become unstable if SLW fails.

A3: We claim that in general, SLW is stable. The reasons are as follows, and we will add them into the revised paper.

1. The similarity between adjacent layers. SLW just simply finds some recovery components to mimic the output between adjacent layers, and as shown in Figure 2 in the main paper, adjacent layers are quite similar. This phenomenon has been verified on larger models like LLaMA2-70B or other models like Mistral-7B in the previous works [2,4].

2. The simplicity of optimization loss. The SLW stage just utilizes simple standard L2 regression loss on the output of adjacent single layers, whose optimization is empirically observed as quite stable. In the experiment, we just used about 10% of the data for the SLW stage (Line 341-343) and it’s already enough for all the layers’ SLW stage to converge.

3. Empirical results support the claim. The final result, like Table A2 in the ‘Reply to all reviewers’ part or Table 1 in the original paper, show SLW stage alone recovers a lot compared to vanilla Direct Sharing. And the computation cost of SLW is also significantly smaller than the SFT stage since we just do independent single-layer fittings than processing the entire large model.

4. Robustness of recovery dataset. The SLW stage is even robust to the choice of recovery dataset. In Table R3-1 in the reply to reviewer tUXM, we show that using GPT4-Alpaca as recovery data can still recover model perplexity, for both SLW and SFT stages. This also shows the robustness of the SLW stage.

Q4: In Table 2, it is described that the comparison is between the model after only SLW, only SFT, and the SHARP algorithm, but there seem to be no results for when only Stage 2 was applied.

A4: In Table 3 in the original paper, we do an ablation study about different experimental settings, and ‘Supervised Fine-Tuning (10%)’ and ‘‘Supervised Fine-Tuning (100%)’ means only Stage 2 is applied. The conclusion shows that SFT is crucial for the final recovery performance, but adding Stage 1 to it can beat Stage 2 alone, especially when the rank of recovery parameters is large. (Section 3.3.3, Line 446-458)

Thank you for your recognition of our paper and your constructive feedback. We have responded to your concerns and will revise our paper based on the discussions. We would also appreciate it if you could let us know if our response addresses your concerns.

Q1: ... we do observe significant performance drop on downstream tasks even after finetuning.

A1: Please refer to ‘All-A3’ in the ‘Reply to all the reviewers’ part. Besides, in ‘All-A2’ we also show that this may be a general problem for all the structural pruning methods, while SHARP has achieved better recovery performance over other structural pruning baselines consistently.

Q2: This leaves me doubt if there's potential overfitting in finetuning process, so that performance on seen datasets are recovered but that on unseen tasks are not … How would the model behave if the finetuning data and the evaluation task does not fully match? For example, will fineutning on GPT4-Alpaca only helps regain the performance on Arxiv-math?

A2: Thanks for pointing out this concern. We conduct the cross-validation experiments you suggested and show the results in Table R3-1 below. We claim that our SHARP algorithm doesn’t have a significant overfitting phenomenon on the recovering dataset.

Table R3-1: Explore if SHARP overfits the finetuning data used by SHARP for recovering performance. We consider the in-distribution task, and focus on the model that is only finetuned on the GPT4-Alpaca. ''using in-distribution data'' means choosing the training data that is from the same dataset as the test task

Here we consider utilizing SHARP with only GPT4-Alpaca data (for both the SLW and SFT stages), and then evaluate the perplexity on different tasks. We can see that

(1) SHARP (with or w/o finetuning) can consistently recover the perplexity on every task, rather than just recover model performance on related tasks like Arxiv-math, supporting that SHARP is not overfitting to the recovery dataset.

(2) Especially, we observe that after the Single-Layer-Warmup (SLW) stage on GPT4-Alpaca, the model perplexity has recovered a lot compared to the vanilla Direct Sharing baseline.

(3) We can also find that for the complete 2-step SHARP, the gap between using in-distribution data (i.e., the standard settings in Table 2 in the main paper, where the training data is from the same dataset as the test task) and GPT4-Alpaca data is not that large, which implicit that our method should have a good generalization in utilizing different recovering data.

Actually, to achieve better perplexity results, we can merge the data from all 5 datasets (While In our paper, we just recover the models on each task one by one for clearer illustration). Thanks again for mentioning this, and we will include this useful ablation study in the revised version.

Q3: …The porposed 2-stage finetuning may also be useful on other model compression techniques. More clarifications on the contribution is needed / Is the proposed 2-stage finetuning scheme limited to layer sharing? Or could it be used for other model compression methods?

A3: Thanks for mentioning this, we claim that:

1. SFT is commonly used in structural pruning methods. As you mentioned, SFT is widely used in the ML community, including structural pruning approaches like LayerPruning[1] nad LLM-Pruner[2].
2. Things like the SLW sound not that novel, but as far as we know, we are the first work to use them in model compression. Actually, SLW just uses the simple L2 regression loss (Line 240-247), and this is a standard choice for fitting something like ‘student-teacher’ settings, where we see the reference layer + LoRA components as a ‘student’ while the target layer as a ‘teacher’. However, we are the first work to try to fit the adjacent layers to save parameters and accelerate inference, therefore as far as we know, we first try to apply this SLW stage to model compression context. If you note we are missing some related works about it, we will also appreciate it very much and will add the citation. We will also add these to revised paper for clearer contribution illustration.

[1] Gromov, Andrey, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. "The unreasonable ineffectiveness of the deeper layers." arXiv preprint arXiv:2403.17887 (2024).

[2] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Llm-pruner: On the structural pruning of large language models." Advances in neural information processing systems 36 (2023): 21702-21720.

Thank you for your recognition of our work and your constructive feedback to help us improve our paper. We will revise our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: More comprehensive comparisons with state-of-the-art efficient inference methods are needed to justify the advantage of the layer-sharing strategy over other categories of strategy, like those mentioned in the paper, e.g., pruning or MobileLLM.

A1: Thanks for your constructive suggestions, we try three more advanced structural pruning methods: LayerPruning, LayerPruning-Adapted, and LLM-Pruner, and show that SHARP performs consistently better than them. Please refer to ‘All-A2’ in the ‘Reply to all reviewers’ part for the details.

Q2: The authors could comment on the advantages of weight sharing in terms of model performance, training cost, memory resources, etc. over other state-of-the-art methods.

A2: Here we mainly compare the structural pruning methods, because methods like quantization are orthogonal to and compatible with our method (as illustrated in A3 as follows and ‘All-A1’ in the ‘Reply to all reviewers’). Compared to LayerPruning, LayerPruning-Adapted, and LLM-Pruner, we can see SHARP has better recovery capability.

Although SHARP may cost slightly more training cost and memory resources due to the existence of the Single-Layer-Warmup (SLW) stage, it costs much less than the Supervised-Fine-Tuning (SFT) stage, which is the necessary stage of all other baselines. The reason is that as illustrated in the paper, SLW is an independent process for different layers, and it just uses a small amount of finetuning data for convergence (like 10%). Also note that the training cost of SLW is just like fitting each single MLP layer, rather than running forward and backward processes on the whole multilayer models, the memory requirement of SLW is quite small. In general, SHARP is a competitive method over the previous advanced structural pruning methods. We will add these discussions to the revised paper.

Q3: In the latency analysis, models are simplified with 4-bit quantization to fit in the iPhone. However, that is not a valid implementation since direct quantization may degrade model performance.

A3: In Table A1 in ‘All-A1’ of the ‘Reply to all reviewers’ part, we validate that deploying 4-bit quantization just results in about 1% downstream performance drop on average, showing that our structural pruning method SHARP is compatible with quantization method.

Q4: On the other hand, it shows that weight sharing is not sufficient to support efficient inference on edge devices.

A4: In ‘OUR POSITION’ in the ‘Reply to all reviewers’ part, we clarify that the main purpose of SHARP is to improve the current structural pruning method and directly accelerate the model inference, instead of aggressively reducing the number of parameters as quantization. These two different directions are always focused on different key points and can be combined, as Table A1 in ‘All-A1’ also shows that SHARP is compatible with the quantization method. Using them together can efficiently accelerate the model inference on edge devices.

I would like to thank the author for proving this general response. The author raised a new claim in this response, stating that the proposed layer sharing is a kind of structural pruning, and "reusing the previous layers will be more efficient than directly pruning them".

Besides my concern on whether raising a new claim in the rebuttal that is never mentioned in the original submission is acceptable, I cannot agree that the claim made by the author is correct.

1. Layer sharing and structural pruning are fundamentally different. Pruning directly removes the layer from the model, so both memory and computation are reduced. This allows the "accelerate model inference without further requirements of hardware design", as said by the author. Layer sharing, on the other hand, only reduces independent weights to be stored, but retains the full computation graph. Even for memory consumption, layer sharing may present more overhead than pruning as the locations of layers being shared need to be flagged. As for computation, the evaluation in the paper utilized "data locality", which is a special feature only available on some specific hardware platforms. Layer sharing cannot be claimed as a general accelerattion method as structural pruning.
2. Reusing should not be more efficient than direct pruning, no matter under what circumstances. The pruned model should have less overhead in model loading, and would have up to 2x less computation compared to the weight sharing model. The benefit of such computation reduction cannot be bridged by utilizing data locality patterns, as there's 0 cost in computing repeated layers for the pruned model.

If the author want to have a fair comparison with structural pruning, the end-to-end wall-clock time should also be reported, preferably on different hardware platforms. A pruned model should be allowed to preserve more layers in a fair comparison to SHARP under the same latency, which may boost the performance of LayerPrune etc. More discussion along this line is needed.

Q4: Reusing should not be more efficient than direct pruning, no matter under what circumstances. The pruned model should have less overhead in model loading, and would have up to 2x less computation compared to the weight sharing model. The benefit of such computation reduction cannot be bridged by utilizing data locality patterns, as there's 0 cost in computing repeated layers for the pruned model.

A4: We agree that when the total number of parameters are the same, then compared to LayerPruning/LLM-Pruner, SHARP is not the most efficient way, since SHARP only reduces the model loading time, but LayerPruning/LLM-Pruner also saves the computation time (in practice, this part of acceleration rate will be lower than 2 times, especially for group-level pruning like LLM-Pruner). However, as in the ‘Important restatement’ part, model loading time takes the majority of latency, so they should share similar inference latency. Besides, note that these two methods have the same amount of parameters after pruning, it’s reasonable to compare them and show that our method has better performance.

Nevertheless, our paper also shows a better replacement strategy than LayerPruning, as mentioned in LayerPruning-Adapted and Table 4 in paper, showing that our paper also improves the previous reusing baseline by providing more fine-grained observation of layer pruning.

Q5: If the author wants to have a fair comparison with structural pruning, the end-to-end wall-clock time should also be reported, preferably on different hardware platforms. A pruned model should be allowed to preserve more layers in a fair comparison to SHARP under the same latency, which may boost the performance of LayerPrune etc. More discussion along this line is needed.

A5: As mentioned in A4, we show that these baselines have the same amount of parameters after pruning, and the close latency due to the domination of model loading time, so it’s still a fair comparison in the same pruning ratio of parameters, which is practically much easier to control compared to achieving the same latency. This is also widely used for the comparison of model compression methods.

More comparison on different hardware platforms can indeed be helpful, but it’s hard to be done in this final rebuttal stage, and as mentioned above, our experiment is reasonable and shows that SHARP has achieved better performance in the same amount of parameters. MobileLLM also mainly do the latency analysis mainly on one mobile platform, so we believe they are beneficial but not necessary experiments.

In short, we thank the reviewer for mentioning these problems, and we will add them in the discussion part in the revised paper, and make our claim clearer for preventing confusion.

[1] Gromov, Andrey, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. "The unreasonable ineffectiveness of the deeper layers." arXiv preprint arXiv:2403.17887 (2024).

[2] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Llm-pruner: On the structural pruning of large language models." Advances in neural information processing systems 36 (2023): 21702-21720.

[3] Liu, Zechun, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong et al. "Mobilellm: Optimizing sub-billion parameter language models for on-device use cases." arXiv preprint arXiv:2402.14905 (2024).

We sincerely appreciate your quick response and pointing out the claims that may be confusing. To answer your question, we first restate the main advantage of SHARP for clarification, we believe that as MobileLLM[3] does, reducing the model loading time is already a significant improvement for accelerating inference in scenarios like mobile phones, and this is the main point we always want to stick to.

Important restatement: SHARP mainly takes advantage of acceleration by directly reducing the time of loading model weights. And in scenarios like mobile devices, This memory loading time can be much more expensive than computation. For example, in the latency analysis (Table 7 in the original paper), we can see that the model loading time occupies 77% of the total processing time. This is also the main idea of MobileLLM[3] to reuse the previous layer (Figure 1(b) in the original paper), i.e., improve the model performance by doubling each layer, while keep the latency increase negligible since the loading time of model weights accounts for the major overhead, and reuse the previous layer almost don’t increase this part of time. Moreover, MobileLLM even shows that ways like layer reusing can doubly reduce the model initialization time, which is also longer than computation time.

And then we answer your concern step by step.

Q1: Layer sharing and structural pruning are fundamentally different. Pruning directly removes the layer from the model, so both memory and computation are reduced.

A1: We mention structural pruning mainly for making it more clear that which baselines are similar to ours and we should compare with. We admit that strictly speaking, SHARP is not the standard ‘structural pruning’ method. The standard structural pruning can not only save the parameters, but also reduce the computation cost directly. We will explain it more clearly in the revised paper. Our main purpose of mentioning is to find the very related pruning methods for suitable comparison, since our ‘structural-pruning-style’ method has similar parameter removing strategies, like removing layers (as LayerSharing[1]) or groups of parameters (as LLM-Pruner[2]), directly, rather than compared with other model compression methods like quantization and sparsification. Sorry again for the confusing statement before

Q2: …Layer sharing, on the other hand, only reduces independent weights to be stored, but retains the full computation graph….As for computation, the evaluation in the paper utilized "data locality", which is a special feature only available on some specific hardware platforms. Layer sharing cannot be claimed as a general accelerattion method as structural pruning.

A2: We agree that the main advantage of SHARP is to reduce the loading time of model weights instead of computation time. However, as shown in the Important restatement, model loading time is much more expensive (77%) than computation time (23%) on mobile devices, so method that improves it has been able to be a general and useful approach for inference acceleration. This is also one of the main supports for MobileLLM[3], which has a very similar latency evaluation as ours.

As for the computation benefit that comes from the “data locality” in SHARP, we agree that this may be related to hardware platforms (like iPhone used in the paper). However, this is not the main improvement we want to stick to. We only briefly mentioned this bonus phenomenon in the latency analysis part.

Q3: Even for memory consumption, layer sharing may present more overhead than pruning as the locations of layers being shared need to be flagged.

A3: We don’t agree that flagging the location of layers will be a significant cost of memory. Here as MobileLLM[3], we at most need to mark something like the index of target layers, which just cost at most KB level of memory, while for the weight of the model can be the level of GB, which is much more than the cost of marking the location.