Search for Efficient Large Language Models

Thanks for the suggestions from the reviewer.

Weakness 1. Potentially biased evaluation

To mitigate the potential biased evaluation problem, we do not use the data from specific downstream tasks (QA datasets such as MMLU or ARC). Instead, for our candidate evaluation and reformation, we use the general dataset WikiText2, which represents generic verified and featured articles on Wikipedia and makes sure that our experiments remain actually zero-shot since no task-specific downstream data is seen during our search. As the model never sees any downstream data (so that it does not know how to be biased), it incurs less biased evaluation for downstream tasks. This setting is consistent with previous works such as FLAP and SliceGPT.

To ensure a fair comparison, we use the same calibration dataset (i.e. WikiText2) for all methods including the baselines. We believe our evaluation is fair and comprehensive. Our method can achieve a better performance on language modeling and QA datasets. Furthermore, FLAP investigates the performance with different calibration datasets (see Appendix C.3 and Table 7 of FLAP). It observes that C4 or WikiText2 as the calibration data, results in a fluctuation of about ±1% in average accuracy for zero-shot tasks, which is not significant.

Weakness 2. MMLU performance

To make a fair comparison, we follow the setting in existing works (such as FLAP and SliceGPT) to use the data from WikiText2 for the candidate evaluation. Besides, it is super efficient to compute the perplexity of a few samples compared with testing the accuracy on the whole test set.

Following the suggestion, we demonstrate the performance on MMLU dataset for our method and the baseline FLAP in Table 9 in the global rebuttal. Our method can outperform the baseline in terms of accuracy on MMLU.

As we mentioned in our response to Weakness 1, we use the general dataset WikiText2 for candidate evaluation, which represents generic text data from Wikipedia and makes sure that our experiments remain actually zero-shot since no task-specific downstream data is seen during our search. Thus, our model is less biased for different downstream tasks.

Weakness 3. QA performance in Table 5

Following the suggestion, we further provide the results on the common sense reasoning datasets for the 50% inheriting ratio in Table 8 provided in the global rebuttal. As observed, our method can lead to lower perplexity and higher downstream evaluation accuracy performance.

Weakness 4. Speed evaluation

Thanks for the suggestion. For the generation speed test, we feed sentences consisting of 64 tokens into the model and adopt float16 with KV cache. The flash attention is not activated. Our results in Figure 7 demonstrate that our smaller models lead to practical inference acceleration. We agree that the inference speed may be different under different evaluation settings. Following the suggestion, we further show the computation cost in GMACs in Table 10 below.

Table 10

Question 1. Writing issues

Yes, the reviewer’s understanding is correct, and "inheriting ratio" = 1 - "sparsity". We provide an explanation for the inheriting ratio in Line 116-117.

Question 2. Annotation

Thanks, we will add the annotation.

Question 3. Clarification

M denotes the number of heads in attention, P denotes the number of channels in MLP (intermediate size). Masks are binary, it should be {0,1} rather than R. We will make it more clear in the revision

3.a. Mask sharing

The mask is shared for the query, key, value, and output projection within a single self-attention module, as well as for the up, gate, and down within a single MLP module, rather than being shared across different blocks. We discuss the mask sharing in detail in the above global rebuttal.

3.b. Align internal computations

As discussed in Line 141-142, for multiple layers inside each module (self-attention or MLP), they have to use the same mask to ensure the aligned internal computations. For example, if the query and key have different masks and thus different/unaligned dimensions, they can not be multiplied elementwise and the attention operation can not be correctly performed. In Appendix A, we demonstrate the mechanism for the self-attention and MLP modules, highlighting that the layers inside the same module need to share the same mask to ensure successful computations. But layers in different modules or blocks do not need to share the same mask. We are sorry for any confusion and will make this more clear in the revision.

3.c. Joint optimization of masks

They are jointly optimized to determine the initial masks as discussed in Line 147-149. Later during our search, all masks are also jointly optimized to satisfy the parameter count constraint.

Question 4. Gamma

$\gamma_{attn}^i$ denotes the inheriting ratio for the $i^{th}$ head in attention. Each attention module can have multiple heads. Thus the inheriting ratio for the attention module is a collection from multiple heads as the inheriting ratio for each head can be different. $\gamma_{mlp}$ denotes the inheriting ratio of the MLP module. It does not have the concepts of multiple heads like the attention module and we just have one single ratio for the whole MLP.

Limitation: Search cost

As we discussed in the global rebuttal, the search cost can be reduced to 20 epochs within 2 hours and is similar to those baselines.

Dear reviewer,

We sincerely appreciate you for spending time reviewing and providing feedback on our paper.

We have submitted one global rebuttal which includes important explanations and additional results following the instructions of one global rebuttal for the paper. However, we find that the global rebuttal is only visible to program chairs and authors. The reviewers are not able to read the global rebuttals. We just raised the problem to the committee and asked if they can fix this issue.

Thanks,

Authors

Thanks for the suggestions from the reviewer.

Weakness 1. Search cost

As we discussed in the global rebuttal, we can reduce the search epoch number from 50 to 20 for LLaMA-7B, which costs 2 hours and can still achieve better performance than other methods. For LLaMA-65B model, we only adopt 20 epochs for search, which costs 5 hours. Besides, compare to other methods, our search cost is similar to those baselines.

Weakness 2. Post training performance

To improve the performance, we adopt an efficient reformation method in Section 3.4 without the need of expensive training or finetuning. As shown in our Figure 6, our reformation can significantly improve the perplexity within a few minutes. We highlight that without training or finetuning, our method already outperforms baselines which are based on recovery training, such as LLM-Pruner and SliceGPT, as shown in our experimental results and Table 8 provided in global rebuttal.

To further evaluate our performance with continual training, we train our searched compact models with LoRA following the same setting of recovery training in LLM-Pruner, which can be finished within another 3 hours for LLaMA-7B. The results are demonstrated in Table 8 provided in global rebuttal. With recovery training, our method can further improve the perplexity and accuracy performance. For more sparse models (50% inheriting ratio) suffering from more significant accuracy loss, recovery training can lead to larger improvements.

Recovery training can further improve the accuracy, requiring more data and computations. Different from recovery training, our reformation can efficiently and effectively lead to better performance than the baselines, with little data and computations. We agree that it is an attractive topic to investigate the pruned model performance with sufficient training. But for LLMs, it is still an open question how to define sufficient training, in terms of model architecture, data quality, data amount, GPUs, and so on. We leave this as our future research.

Question 1. Mask sharing

The weights in the same module of each block share the same mask, but different blocks do not share masks. Each block has its own masks, which are not shared with other blocks. The reviewer’s understanding is correct. We discuss mask sharing in detail in the global rebuttal. Line 141 shows that there are two masks for each block, meaning there will be $2k$ masks for $k$ blocks, as the masks of different blocks are not shared.

Question 2. Fine-tuned compressed model compared to LLM-Pruner

We show the results compared to the fine-tuned LLM-Pruner in Table 8 provided in the global rebuttal. Results show that our method can perform better than LLM-Pruner even without finetuning.

Thanks for the suggestions from the reviewer.

Weakness & Question: Comparison with SparseGPT

The pruning or search granularity of our method and SparseGPT are different, thus they should not be directly compared. Our method searches smaller compact models, which is more like the structural pruning method to remove rows and columns of weight matrices. Different from our structure search, SparseGPT is an irregular pruning method to remove weights without strong constraints (such as, the removed weights should be in the same rows or columns). Thus in the experiments, we compare with other structure pruning methods such as SliceGPT and FLAP, without comparing to SpareGPT. Furthermore, the baseline SliceGPT already provides certain comparisons with SparseGPT, as shown in Table 1 of SliceGPT. As mentioned above, it is hard to make a fair comparison between structure and irregular pruning. In Table 1 of SliceGPT, the sparsity of SparseGPT is 50%, while the sparsity of SliceGPT is 25% or 30%. The results in their Table 1 can serve as a reference, and our method can outperform SliceGPT non-marginally.

Thanks for the suggestions from the reviewer.

Weakness 1. Difference from network pruning

Our proposed search method significantly differs from standard neural network pruning techniques. Specifically, our method includes a search initialization process to identify the effective initial architecture and an evolution architecture search with special mask mutation and efficient candidate evaluation, which are not typical in standard weight pruning techniques. We are the first to incorporate the architecture search technique for LLMs to find efficient architectures. There are no other search baselines and thus we mainly compare with pruning works. Although our method significantly differs from traditional pruning, both our search and pruning can lead to efficient lightweight LLMs.

Weakness 2. Reformation

Thanks for the suggestion, we will explain more about the reformation in the abstract and introduction.

Weakness 3. Search cost

We highlight that our search cost is just similar to the baselines. Details can be found in the above global rebuttal.

3.a. Performance as a function of search cost.

Figure 5 clearly demonstrates how the proposed method performs as a function of the search budget (number of epochs and thus search hours). As we discussed in the global rebuttal, we can reduce the search cost without significantly sacrificing the performance. Note that, with 20 epochs, for LLaMA-7B, we can still achieve 6.23 and 7.21 perplexity on WikiText2 under 90% and 80% inheriting ratios, respectively, which are better than all baselines shown in Table 2.

3.b. Lower cost for memory and data in the search

Besides the training time on GPU to finish the search, our method achieves a better performance with less memory and data. (a) Compared to LLM-Pruner, which identifies weight importance using backward passes, our method relies solely on inference. This significantly reduces GPU memory consumption, enhancing efficiency. (b) Compared to the SliceGPT, it requires 1024 data samples for calibration. Our method requires significantly fewer calibration samples (128) for weight optimization after the search, leading to better performance on various datasets.

Weakness 4. Quantization

Quantization methods are not direct competitors but rather complementary to our search approach. Existing quantization methods can be applied to the subnets identified by our method, further enhancing acceleration through integer matrix multiplications. In practice, quantization methods require specialized computation kernels to handle integer matrix multiplications, which can introduce additional computational overhead and complexity in implementation, particularly during inference. This can pose challenges when deploying quantized models on diverse hardware platforms which may not support the quantized model. Our approach, on the other hand, maintains the original computation framework of LLMs, without the need for additional operations or specialized implementations.

Regarding the performance hit, we would like to clarify that the results in Table 7 were generated with a sequence length of 128, rather than 2048, for sequence length ablation study. Given that the dense LLaMA-7B model achieves the perplexity of 12.62 at 128 sequence length, our method, with a 90% ratio, results in the perplexity of 13.40. This indicates a relatively small performance gap. With a larger sequence length such as 2048, the performance of all methods can be better and we can still achieve the best performance as shown in Table 2 and Table 3.

Question 1. Search cost and model size

We discuss the search cost in detail at the global rebuttal. For LLaMA-7B, we still can achieve better performance than other methods when searching only with 20 epochs within 2 hours. For LLaMA-65B, we only adopt 20 epochs to get the subnets, which takes 5 hours.

Question 2. Search cost with 5 hours

As we mentioned in global rebuttal, for LLaMA-7B on one A100 GPU, it takes 5 hours for 50 epochs during the search process. The search already converges at 20 epochs as demonstrated in Figure 5, which only takes 2 hours and already achieves better performance than other methods. Besides, for large models such as LLaMA-65B, we only adopt 20 epochs for faster search, which takes just 5 hours.