Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models

[Q1] Approach is restricted to encoder-based models but many of the current LLM approaches are decoder-based.

[A1] Theoretically, Kprune is applicable to the decoder-based models since decoder-based Transformers also consist of multi-head attention (MHA) and feedforward network (FFN) sub-layers; we are able to measure the amount of knowledge in each attention head and neuron. In this paper, we focus on encoder-based models to exhaustively demonstrate the effectiveness of Kprune since there are both state-of-the-art retraining-free and retraining-based algorithms for encoder-based models; we demonstrate the accuracy of Kprune by comparing it to retraining-free pruning algorithms and show the efficiency of Kprune by comparing it to retraining-based algorithms. On the other hand, it is impossible to compare Kprune to the retraining-based pruning algorithms using LLMs since retraining-based pruning algorithms require intractable time to prune LLMs. As we mentioned in the Conclusion, extending Kprune to decoder-based models is an interesting future work.

[Q2] Sec 1: What does the "%p" notation mean?

[A2] “%p” is a symbol of percent-point which is used for representing the difference of percentage values. The term percent-point is distinct from percent (%) though both terms signify shifts in percentage values. Percent (%) expresses the proportional change between two numbers, while the percent-point represents the absolute difference. For instance, a 10% increase from 50% results in 55%, whereas adding 10 percentage points (%p) to 50% yields 60%. We insert a footnote in Section 1 about %p for clarity.

[Q3] Sec 2.2: What does 'established' mean in this context?

[A3] In this context, ‘the mask variables are established’ means that the values of the mask variables are determined. We aim to deliver that imposing a mask variable as zero is equal to the pruning of the corresponding attention head or neuron since the output of MHA or FFN is represented as the masked summation of attention heads or neurons, respectively. We substitute the word “established” with “determined” for clarity.

[Q4] Equation 7: The LHS $K_{rep,l}(X_{\mathcal{T},l},X_{\mathcal{S},l};0)$ should be a function of $i$.

[A4] We use the notation $K_{rep,l}(X_{\mathcal{T},l},X_{\mathcal{S},l};0)$ as the simplified notation of $K_{rep,l}(X_{\mathcal{T},l},X_{\mathcal{S},l};m_{l,i}=0)$ which is a function of $i$. We substitute the simplified notation into $K_{rep,l}(X_{\mathcal{T},l},X_{\mathcal{S},l};m_{l,i}=0)$ to clearly show that $K_{rep,l}$ is a function of $i$ in Sections 3.2 and B.1.

[Q5] Sec 3.3: "We estimate the importance score not only in the target sublayer but also in the sublayers above the target sublayer" Why is it necessary to estimate the importance score for sublayers above the target sublayer considering that only the heads/neurons in the target sublayer are pruned when a given layer is considered i.e. From the description/Figure 2, it seems like Algo 1 is run for each sub-layer.

[A5] Kprune measures the importance scores of the target sublayers and upper sublayers to determine the number of attention heads and neurons to prune in each sublayer, considering their global importance. If we do not measure the importance scores of masked units (attention heads or neurons) in upper sublayers, we cannot estimate how important the target sublayer is in the entire model. Therefore, we have to prune each sublayer equally, without considering the sublayer-wise importance, resulting in a significant accuracy drop. In Figure 2, our target sublayer is the 2nd sublayer and we use the importance score of the 3rd and the 4th sublayers to determine the amount of neurons to prune (b). As illustrated in (c) we prune neurons only in the 2nd sublayer (target), not in the 3rd and the 4th sublayers even though we measure the importance of masked units in the 3rd and the 4th sublayers. Then, we move on to the next iteration to prune attention heads in the 3rd sublayer. We revise the description of Figure 2 in Section 3.1 to clarify the reason why we estimate the importance scores in the upper sublayers.

[Q6] The algorithm performs sublayer-wise pruning from the bottom to the top sub-layer. If we refer to this as a single pass, is it ever necessary in practice to do a second pass to achieve the desired number of FLOPs?

[A6] You don’t need to manually tune hyperparameters or re-run Kprune to match the desired number of FLOPs. Kprune automatically finds the global pruning configurations that match the pre-defined desired pruning rate by adjusting the FLOPs budget after pruning each sublayer. We revise the last sentence in Section 3.4 to clearly deliver that Kprune does not require user intervention, such as hyperparameter tuning, to match the desired number of FLOPs.

We sincerely appreciate the reviewer’s careful feedback. We respond to your feedback below.

[Q1] The writing of the introduction section seems unreasonable. I think some challenges and other content should be written in the introduction instead of the method.

[A1] Thank you for your constructive review. We have revised the Introduction section to reflect your comment. We add a paragraph about the need for retraining-free pruning algorithms and clarify the limitations of existing retraining-free algorithms.

[Q2] In the main experiment table (Table 1), there are not enough baselines for comparison.

[A2] To the best of our knowledge, Kwon et al. (NeurIPS’22) and KCM (ICML’23) are the best-performing retraining-free pruning algorithms for encoder-based models, and Kprune is the first algorithm that outperforms both. In detail, Kwon et al. is the first retraining-free pruning algorithm for PLMs and Kwon et al. is the only competitor for KCM. There are a few retraining-free algorithms for (decoder-based) LLMs, such as SparseGPT[1] or Wanda[2], but, they employ an impractical unstructured pruning pattern; they require inefficient sparse tensor representations for memorization and customized hardware for acceleration.

[Reference]

[1] Frantar, Elias, and Dan Alistarh. "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot." (2023).

[2] Sun, Mingjie, et al. "A Simple and Effective Pruning Approach for Large Language Models." arXiv preprint arXiv:2306.11695 (2023).

We sincerely appreciate the reviewer’s careful feedback. We respond to your feedback below.

[Q1] While not a direct comparison, it would be interesting to compare KPrune's performance to stronger training-based methods like CoFiPruning[1]. KPrune's major advantage is requiring much less time for pruning, though training-based approaches may achieve better end results.

[A1] Thank you for suggesting an interesting experimental setup. However, as we mention in the “Competitors” paragraph in Section 4.1 of our paper, there is a reason for excluding CoFi as a competitor. According to Kwon et al.[2], CoFi shows similar or worse accuracy than DynaBERT (Figure 6 in Kwon et al.) while requiring more than 2.7 times longer time for pruning than DynaBERT (Table 2 in Kwon et al.). Therefore, we chose DynaBERT and EBERT as our competitors and conducted our experiments accordingly.

[Q2] Since BERT models are relatively small in scale nowadays, reducing training compute is less impactful compared to large language models. It would be interesting to evaluate if KPrune can effectively scale up to larger and more powerful models where saving compute will be more significant.

As noted in the response [A2] for the reviewer [akzo], the cost of Kprune for a model with $L$ sublayers is equivalent to performing $(L+1)/2$ times of forward and backward propagations on tiny sample data. If we follow the setting of SparseGPT[1], an “unstructured” pruning algorithm for large language models (LLMs), this involves using only about 128 sample instances. Therefore, if the hardware setting is adequate for running inference on massive LLMs, it is entirely feasible to execute Kprune within a tractable amount of time. Extending accurate and efficient Kprune to massive LLMs is our promising future work.

[Reference]

[1] Xia, Mengzhou, Zexuan Zhong, and Danqi Chen. "Structured pruning learns compact and accurate models." Annual Meeting of the Association for Computational Linguistics (2022)”: 1513-1528

[2] Kwon, Woosuk, et al. "A fast post-training pruning framework for transformers." Advances in Neural Information Processing Systems 35 (2022): 24101-24116.

We sincerely appreciate the reviewer for expressing your concerns. We respond to your concerns and questions below.

[Q1] The paper focuses specifically on encoder-based language models, so it may not be applicable to other types of language models.

[A1] Theoretically, Kprune is applicable to the decoder-based models since decoder-based Transformers also consist of multi-head attention and feedforward network sub-layers; we are able to measure the amount of knowledge in each attention head and neuron. On the other hand, we focus on encoder-based models to exhaustively demonstrate the effectiveness of Kprune since there are both state-of-the-art retraining-free and retraining-based algorithms for encoder-based models; we demonstrate the accuracy of Kprune by comparing it to retraining-free pruning algorithms and show the efficiency of Kprune by comparing it to retraining-based algorithms. On the other hand, it is impossible to compare Kprune to the retraining-based pruning algorithms using LLMs since retraining-based pruning algorithms require intractable time to prune LLMs. As we mentioned in the Conclusion, extending Kprune to decoder-based models is an interesting future work.

[Q2] The authors do not provide a detailed analysis of the computational resources required to implement Kprune, which could be a potential limitation for some applications.

[A2] The computational cost of Kprune is composed of the computational costs of (a) Knowledge measurement, (b) Knowledge-preserving mask search (KPMS), and (c) Knowledge-preserving weight-tuning processes (KPWT). During (b) KPMS, the process involves inexpensive calculation of importance scores based on the measured amount of knowledge, and comparison of the scores to find the pruning masks; thus KPMS takes a very short time. Moreover, (c) KPWT is also an inexpensive process, completing within a second for all sublayers when using the “lstsq” solver provided in PyTorch. Therefore, the computational cost of Kprune mainly depends on the (a) knowledge measurement process, which involves repetitive forward and backward operations of sublayers.

If we assume that $M$ is the number of operations required for forward and backward propagations in a model with $L$ sublayers, then the operations required per sublayer are considered as $M/L$. The knowledge measurement process includes forward and backward operations for each sublayer from the bottom to the top. Each operation includes $L$, $L-1$, ..., and $1$ times of forward and backward operations of sublayers, and requires $(L+1)/2 * M$ operations in total, i.e. it is the same as the cost of running $(L+1)/2$ epochs on a sample dataset. Combined with the fact that Kprune utilizes a tiny sample dataset (about 0.64% of the MNLI dataset), the cost of Kprune is significantly low and Kprune is scalable to the larger language models. In our paper, Kprune takes a few minutes on a single NVIDIA 1080Ti GPU for pruning BERT-base which is up to 422 times shorter time for pruning retraining-based algorithms.

[Q3] The paper does not apply to large language models which are commonly used now, experiments on larger models are expected. Can the proposed technique applied to LLMs?

[A3] As noted in [A2], the cost of Kprune for a model with $L$ sublayers is equivalent to performing $(L+1)/2$ times of forward and backward propagations on a tiny sample dataset. If we follow the setting of SparseGPT[1], an “unstructured” pruning algorithm for large language models (LLMs), this involves using only about 128 sample instances. Therefore, if the hardware setting is adequate for running inference on massive LLMs, it is entirely feasible to execute Kprune within a tractable amount of time. Extending accurate and efficient Kprune to massive LLMs is our promising future work.

[Reference]

[1] Frantar, Elias, and Dan Alistarh. "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot." (2023).