Enhancing One-Shot Pruned Generative Pre-training Language Models through Sparse-Dense-Sparse Mechanism

Dear Reviewer XgFT,

We sincerely appreciate your thorough analysis and insightful comments on our manuscript, which have provided us with valuable perspectives for enhancing our manuscript. We have responded to each of your questions and provided detailed explanations and additional data to support our findings.

Response to weakness 1: Thanks so much for your suggestion, Wanda is a very suitable one-shot pruning method as our baseline. Therefore, we construct SDS optimization experiments with Wanda as the basic pruning method.

In the comparative analysis, SDS shows a significant improvement over Wanda, with an average perplexity of about 33.33, which is lower than Wanda's 42.79. In the downstream tasks, SDS also obtains an average performance improvement of 3.1% compared to Wanda. The above results illustrate that SDS enables the model performance to be improved.

In fact, the Wanda method is less effective than SparseGPT for pruning undersized (non-redundant / more fully trained) PLMs. On the one hand, the idea of Wanda to protect outlier activations is less effective on undersized PLMs with insignificant outliers. On the other hand, Wanda fails to compensate for the pruning error as SparseGPT although it can find an effective sparse mask.

A brief introduction to Wanda, experimental results, and more analysis can be found in Section A.5. Additional comments are provided for your convenience to refer to the experimental results.

Response for weakness 2: As you mentioned, LLaMA is a model that is difficult to compress, and constructing experiments using LLaMA as a base model is better for exploring the validity of SDS.

We take the 7b-sized models of LLaMA and LLaMA2 with 2:4 sparse as the pruning configuration and SparseGPT/Wanda as the base pruning method to verify the language modeling perplexity and multi-task performance of the SDS-optimized pruned models.

Experimental results show an average perplexity reduction of approximately 10.90% and an accuracy increase of around 2.46%. These findings demonstrate the SDS's effectiveness in enhancing larger and stronger pruned PLMs' performance.

For the LLaMA model, the language comprehension perplexity is impaired on a doubling scale after one-shot pruning with SparseGPT or Wanda, proving that LLaMA is indeed a PLM that is difficult to compress. LLaMA, being trained on large-scale data, is considered a non-redundant model even though it already has quite a few parameters. The fact that pruning of redundant parameters is somewhat unachievable on LLaMA elevates the difficulty of model compression and the need to perform SDS optimization.

The detailed experimental results and analysis can be found in Section A.5. Additional comments are provided for your convenience to refer to the experimental results.

Response for weakness 3: As the volume of PLM grows, the difficulty and inadequacy of its training grows rapidly. In the case of OPT-2.7b, its redundancy will be much higher than that of the smaller version OPT-125m, such that the implementation of one-shot pruning alone will already ensure very little degradation of its performance. Taking the premise that the theoretical upper bound on the performance of the SDS-optimized pruned model is on par with its dense counterpart, the effectiveness and necessity of performing SDS optimization for a highly redundant PLM is limited. However, as training techniques and high-quality data continue to be upgraded, newly born large-scale PLMs will increasingly exhibit low-redundancy characteristics, such as those of LLaMA. The SDS optimization will show more valid superiority ahead.

Response for question 1: Keeping the calibration set constant, and iteratively performing the sparse-dense-sparse optimization will no longer improve the performance of the pruned PLMs because of the overfitting phenomenon that arises. Better performance will result if different data is used for each iteration. Even so, we still recommend using only a single SDS to avoid potential overfitting phenomena. Specific experimental and analytical procedures can be found in Section A.4.

In conclusion, we hope that our responses adequately address the concerns and suggestions you have raised. We are committed to enhancing the quality and impact of our research and believe that your feedback has significantly contributed to this goal. We look forward to any further suggestions or considerations arising from this discussion.

Thank you once again for your valuable and insightful review.

Best regards,

Authors 5070.

The pruning configuration is set to 2:4 sparsification.

The above table contains the experimental results with OPT/GPT as the base model and SparseGPT/Wanda as the base pruning method. Accuracy (%) is obtained by zero-shot evaluation and averaging over seven downstream tasks, including COPA, Lambada, OpenbookQA, PIQA, RTE, StoryCloze, and Winogrande.

The above table contains the experimental results with OPT/GPT as the base model and SparseGPT/Wanda as the base pruning method.

Dear Reviewer k1aZ,

Thank you for your incisive observations and questions regarding our work. We have addressed all your queries, offering comprehensive clarifications and supplementary data to reinforce our manuscript.

Response to weakness 1: The process of running SDS is efficient and memory-friendly: (i) only the layer being optimized needs to be in GPU memory; (ii) optimization between layers is parallelizable, so the device's power can be fully utilized for speedups. The following table shows the runtime of SDS for single-device serial, multi-device (our implementation, on 8 32GB Nvidia V100 GPUs) parallel, and extreme parallel (theoretical) cases (see Section A.5 for more analysis).

To highlight the contrast in outcomes, we introduce additional experimental comparisons including GPT, LLaMA as the base model, and Wanda as the base pruning method, and calculated the average accuracy on all downstream tasks (Table 6, Table 7, and Table 9 in Section A.5). Experimental results demonstrate that SDS outperforms SparseGPT and Wanda by decreasing language comprehension perplexity by an average of 6.4 and achieving an average accuracy improvement of 1.9% across multiple downstream tasks. Thus, the SDS pruning method can be proven effective.

Additional comments are provided for your convenience to refer to the experimental results.

Response to weakness 2: In the revised manuscript, we included experiments involving the larger PLM LLaMA. Our findings demonstrate that SDS improves LLaMA's accuracy by 2.5% when compared to SparseGPT/Wanda. However, as the model size increases, redundancy also grows, allowing SparseGPT to achieve sufficiently good results independently. In such cases, SDS may not be necessary. However, for more recent models like LLaMa, which outperform OPT at similar sizes—highlighting LLaMA's compactness—SDS can still effectively enhance the performance of pruned PLMs.

Response to weakness 3: Thanks for your interest in practical inference speedups for our pruned PLMs. Our approach is aimed at optimizing the performance of pruned PLMs under 50%/2:4/4:8 sparse configurations. The real-time speedup is contingent upon the support provided by sparse acceleration libraries or hardware capabilities. An instance of this is demonstrated by DeepSparse [1], showcasing a round 20% end-to-end speedup achieved through 50% unstructured sparsity on CPU. Nvidia [2] also demonstrates that a 2:4 sparse GEMM achieves an ideal twofold performance increase compared to an equivalent dense GEMM, leveraging Sparse Tensor Cores on GPUs with the Ampere architecture. Currently, 4:8 sparse acceleration is not supported. However, the experimental results in the manuscript will hold valuable implications for guiding future hardware designs.

[1] DeepSparse: https://github.com/neuralmagic/deepsparse

[2] Mishra A, Latorre J A, Pool J, et al. Accelerating sparse deep neural networks[J]. 2021.

Response to question 1: Thank you for pointing out the error in our formula. We have corrected it as follows: $X\leftarrow{W}_i^{\text{sparse-2nd}}X$ in the revised manuscript. We confirm that this correction does not impact our results or conclusions.

Response to question 2: You are correct that we use the same samples throughout each SDS step. We appreciate your interest in using different samples within SDS, and the corresponding result is detailed in row 11 of Table 4. The efficacy of employing different samples within SDS closely mirrors that of the standard SDS, with only two tasks surpassing it. In the short term, this may be caused by the failure to learn sufficiently about each batch of samples as well as the varying quality of the samples. This finding confirms the advantage that SDS does not need to rely on additional samples.

Response to question 3: Yes, you are right. We intended to convey that our method utilizes only a small amount of unlabeled data. Acknowledging that the original phrasing (We did not use a large amount of labeled data) may have caused confusion, we have reworded the expression to be directly affirmative. We believe this change maintains the accuracy of our presentation while improving readability.

In summary, we trust that our responses have thoroughly addressed your review points. Your valuable feedback has significantly helped refine our research. We welcome any further discussion or clarification if needed.

Thank you for your invaluable contributions to our work.

Best regards,

Authors 5070.

The above table contains the experimental results with OPT/GPT as the base model and SparseGPT/Wanda as the base pruning method. Accuracy (%) is obtained by zero-shot evaluation and averaging over seven downstream tasks, including COPA, Lambada, OpenbookQA, PIQA, RTE, StoryCloze, and Winogrande.

The above table contains the experimental results with OPT/GPT as the base model and SparseGPT/Wanda as the base pruning method.

The pruning configuration is set to 2:4 sparsification.

Dear Reviewer hYrm,

Thank you for your valuable feedback on our submission. We appreciate the time you have dedicated to reviewing our work and offering constructive comments. Your insights are particularly helpful in understanding the implications of our pruning methods for both undersized and larger pre-trained language models and validating the impact of performing SDS optimization iteratively.

In response to your concerns, we would like to provide some clarifications and insights based on our in-depth exploration and experiments.

Response to weakness 1: We chose the Llama model to further validate the effectiveness of the SDS framework on larger and stronger PLMs. We found that when pruning the model using conventional pruning methods (SparseGPT/Wanda) incurs a large performance loss, SDS is effective in optimizing the pruned model. Below are the experimental results and analysis, you can also find them in the second part of Section A.5 (Page 19 in our updated manuscript).

We take the 7b-sized models of LLaMA and LLaMA2 with 2:4 sparse as the pruning configuration and SparseGPT/Wanda as the base pruning method to verify the language comprehension perplexity and multi-task performance of the SDS-optimized pruned models, main results are shown below:

The above results highlight that SDS achieves an average perplexity reduction of approximately 10.90% and an accuracy increase of around 2.46% over other one-shot pruning methods. This substantial improvement underscores the effectiveness of SDS in enhancing the performance of larger and stronger pruned PLMs.

Response to question 1: The iterative implementation of the sparse-dense-sparse optimization failed to achieve better performance for the model, which we attribute to the overfitting phenomenon caused by overlearning the same batch or distribution of samples. In fact, optimizing with different samples in each iteration can produce higher-performance models. At the same time, we also found that SDS benefited more from multiple samples than SparseGPT.

Specific experiments and more analysis can be found in Section A.4 (Page 17 in our updated manuscript).

Response to question 2: SDS significantly enhances the performance of pruned undersized PLMs (OPTs) and larger PLMs (Llama). These models are characterized by low redundancy and they can be fully trained or trained on a sufficient number of samples. Conversely, models with notable redundancy, often comprising inadequately trained massive PLMs, can already be pruned by SparseGPT and Wanda with minimal or no loss. Consequently, the necessity to apply SDS and its optimization diminishes in such cases. SDS excels at efficiently optimizing pruned non-redundant PLMs, regardless of the model's volume (even though non-redundant PLMs often appear as undersized ones). As training techniques and high-quality data continue to be upgraded, newly born large-scale PLMs will increasingly exhibit low-redundancy characteristics, such as those of LLaMA. The SDS optimization will show more valid superiority ahead.

In summary, we trust that our clarifications and additional data have effectively addressed your valuable feedback. We deeply appreciate the role your review has played in sharpening our research and contributing to its depth. We remain open to any additional insights or recommendations you may have.

With gratitude for your thorough and constructive critique,

Authors 5070.