Dear Reviewer VeXf ,

Thanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would appreciate it if the reviewer considers raising the score. Please see our responses below one by one：

Q1: Compare to existing AutoML techniques.

A1: In contrast to common AutoML and evolutionary algorithms, our method is specifically tailored for sparsity allocation discovery in LLMs and involves several novel components:

(1) Novel problem formulation: We are the first to frame LLM sparsity allocation as an AutoML problem, opening new avenues for optimizing LLM efficiency.

(2) Tailored search space: We introduce a distinctive search space customized for LLM sparsity allocation, combining pre-processing, reduction, transformation, and post-processing operations in novel ways, allowing for more nuanced and effective sparsity distributions.

(3) Function discovery and generalization: Diverging from typical AutoML methods such as NAS and HPO that search for specific modes or hyperparameters, our framework emphasizes generalized function discovery, identifying common patterns across LLMs and formuling interpretable sparsity allocation.

(4) Search acceleration: We develop LLM-specific acceleration techniques to reduce search time, making our DSA practical for large-scale LLM optimization.

Q2: Compare to ECoFLaP [ref1] .

A2: We clarify that our novelty lies in being the first automated search for adaptive sparsity methods, which significantly differs from traditional adaptive pruning methods like ECoFLaP. Our work differs significantly from ECoFLaP in several key aspects：

(1) Automation and adaptability: We employ an automated search method that eliminates the need for expert design and adapts strategies to different models and tasks, whereas ECoFLaP relies on hand-designed, hyperparameter tuning.

(2) Comprehensive search space: Our comprehensive search space systematically maps element-wise scores → per-layer importances → sparsity ratios. In contrast, ECoFLaP simply computes the keep ratio linearly during its two-stage pruning.

(3) Superior performance: Our method obtains significant performance gains across various large language and multimodal models, demonstrating superior performance compared to ECoFLaP (see Table below).

Table : Perplexity of Wanda, ECoFLaP and our DSA with LLaMA 7B at 0.6 sparsity on WikiText2

We will augment this discussion and cite more Efficient AI studies [ref1]-[ref5] in the revision.

Q3 & Q5: About computational resources and cost compared to traditional methods.

A3: Our responses are:

(1) Our method's main computational cost is in the initial search phase, taking about 0.5 days on LLaMA-1-7B. But the discovered allocation functions are transferable to other models without additional costs. This one-time cost can be spread across multiple pruning runs. As discussed in the limitations section. we've developed acceleration techniques to address the search cost challenge common in AutoML methods.

(2) Most conventional pruning methods also require time-consuming hyperparameter tuning, which may be less efficient than our well-optimized automated search process.

(3) Applying our allocation function to prune the model proves highly efficient (see following Table for detailed pruning speeds). This efficiency stems from two key factors: (a) we utilize element-wise scores from the uniform pruning method directly, avoiding extra forward and backward computations. (b) we employ reduction operations to simplify computations linearly.

Table : Comparison of time overhead used for computing the pruning metric of LLaMA-1-65B to 50% sparsity.

Q4: About significance of results.

A4: Our responses are:

(1) We would like to clarify that our method consistently achieves groundbreaking gains across multiple challenging tasks (reasoning and multimodal benchmarks). Notably, the LLaMA-1|2|3 model pruned by our DSA reach 7.48%|5.69%|14.14% gains on the state-of-the-art models Wanda and SparseGPT.

(2) To ensure fair comparisons, we follow standard settings like 50% sparsity, where baseline results are strong, making additional gains challenging.

(3) Our experiments in Table 8 at higher sparsity levels (65%-80%) on LLaMA-1 show our method's ability to achieve substantial gains.

(4) Results in the rebuttal PDF (See our general response) further demonstrate that our method can achieve substantial gains, with improvements ranging from 1% ~ 7% at 60% sparsity, 2% ~ 7.6% at 70% sparsity, and 1.4% ~ 4.2% at 2:4 sparsity.

References:

[ref1] ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models. In ICLR, 2024.

[ref2] Training Neural Networks with Fixed Sparse Masks. In NeurIPS, 2021.

[ref3] Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. In ICLR 2024.

[ref4] LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning. In NeurIPS 2022.

[ref5] VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks. In CVPR 2022.

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.

Dear Reviewer VeXf ,

We express our deepest appreciation for your careful and constructive feedback. Our rebuttal addresses your concerns comprehensively, and we welcome any additional questions. If our response has successfully addressed your concerns and clarified the significance of our work, we would be immensely grateful if you could reconsider your recommendation. We promise to add citations to related studies [ref1]-[ref5] in the revision. We have diligently integrated feedback from all reviewers and hope this is positively reflected in your evaluation. Thank you for investing your valuable time in reviewing our response.

With profound respect and gratitude,

Paper 66 Authors

Dear Reviewer VeXf ,

We would like to sincerely thank you again for your valuable feedback and insights, which have greatly improved our paper. We promise to thoroughly reflect all your comments in the final manuscript. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be immensely grateful if you could reconsider your recommendation.

Best regards,

Paper 66 Authors

(d) In our A5 for Reviewer of38, we apply our allocation functions to ConvNeXt in Table below. surpasses other methods, especially at higher sparsity levels, showing its generalizability for various models (Recognized by Reviewer of38 with improving the score).

Table: Accuracy (%) of Sparse ConvNeXt-Base on ImageNet-1K.

(3) About different sparsity levels: Yes. We directly transfer searched allocation function (see Equation 6) for different sparsity levels and achieve noticeable gains at 65%-80% sparsity on LLaMA-1 in Table 8. In the rebuttal PDF (See our general response), our searched allocation function demonstrates gains ranging from 1.24% to 7.03% at 60% sparsity and 1.91% to 7.68% at 70% sparsity on LLaMA-2 and on LLaMA-3.

Q8: About more comparison with ECoFLaP across various models, sparsity levels, and datasets.

A8: Following the suggestion, we conduct more experiments to compare our DSA method with ECoFLaP across various models, sparsity levels, and datasets. The results, summarized in Table below, demonstrate the superior performance of our DSA method across different LLaMA models and sparsity rates. For the LLaMA-2-7B model, DSA achieves gains of 1.92% and 1.18% at 60% and 70% sparsity levels respectively, compared to ECoFLaP's gains of 0.77% and 0.52%. Similar trends are observed for the LLaMA-2-13B model, where DSA outperforms ECoFLaP with improvements of 1.72% and 1.24% at 60% and 70% sparsity levels, surpassing ECoFLaP's gains of 0.86% and 0.65%. The most significant improvements are seen in the LLaMA-3-70B model, where DSA achieves remarkable gains of 2.23% and 2.34% at 60% and 70% sparsity levels, substantially outperforming ECoFLaP's gains of 0.71% and 0.68%. These consistent improvements across different model sizes and sparsity levels highlight the effectiveness and generalizability of our DSA method, demonstrating its superiority over ECoFLaP in enhancing the performance of pruned language models.

Table: Mean accuracies (%) of our DSA on zero-shot task with 7 different datasets.

---

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We promise to include these discussions and citations in the revision and sincerely hope that reviewer will consider improving the recommendation.

Dear Reviewer VeXf ,

We would like to sincerely thank you again for your constructive feedback and apologize for any unclear points in the initial responses. After receiving new comments, we have made our best efforts to augment experiments and clarifications as follows. Please see our responses below one by one：

---

Q6& Q7: Generalizability of allocation function and transferability between sparsity levels.

A6: We would like to highlight that all of our performance gains across models (LLaMA-1|2|3, Mistral, Vicuna, and OPT) and tasks (reasoning and multimodal benchmarks) are obtained by transferring the same searched allocation function on LLaMA-1-7B (see Equation 6), without additional individual searches:

(1) About empirical evidence: (a) As detailed in Table 2 &3 &4 & 5 [Line 250-286], our searched allocation function demonstrates high generality and consistent improvements across various model sizes and architectures, including LLaMA-1, LLaMA-2, LLaMA-3, OPT, Mistral, and LLaVA (Vicuna-based) models. For zero-shot accuracy at 50% sparsity in Table 2, our searched allocation function shows significant gains, improving magnitude pruning by up to 14.14%, Wanda by up to 4.36%, and SparseGPT by up to 9.82% on LLaMA-3 8B. On the MMLU task in Table 4, our searched allocation function shows 0.96% gains over OWL for LLaMA-1 13B. In multimodal tasks in Table 6, our searched allocation function improves performance by up to 1.65% on LLaVA-1.5 7B and 1.98% on LLaVA-1.5 13B (SQA task) at 50% sparsity. (b) We also directly transfer our searched allocation function on LLaMA-3 70B in the rebuttal PDF (See our general response). Our searched allocation function shows 2.23% to 4.38% gains across different pruning approaches, aligning with our effectiveness on multiple LLMs. These results underscore the effectiveness and generality of our searched allocation function across various models, tasks.

(2) Understanding generalizability: Sufficient experimental evidence in (1) solidly demonstrates the generality of our searched allocation function across different models and tasks. We provide several aspects to understand its generality: (a) In the model sparsity area, different LLMs share the common sparsity laws (e.g., important weights have salient gradients) resulting in sparsity methods with fixed metrics or functions that can be generalized to various models. For example, SparseGPT, Wanda, OWL and ECoFLaP also apply the same metric for different tasks without discussing whether their functions need to be changed for different structure and depth models (apologize if I missed). Our allocation function search space is based on observation element-wise score distributions in LLMs, as discussed in the Introduction [Lines 62-78] and Figure 1(left). This observation aligns with OWL (see Figure 1 (middle)) and the sparsity laws that the initial layers of LLMs are more important. Because of these reasons, our searched allocation function naturally follows the sparsity laws, allowing good generalizability to the different LLMs. (b) The operators contained in our allocation function are architecture-agnostic and serve to normalize and eliminate architecture variance. For example, our pre-process operator standardizes inputs by normalizing scores across layers, ensuring consistent performance metrics by addressing scale variations. As our stability analyses [Lines 243-250] and theoretical understanding in Appendix B [Lines 554-571], our allocation function is stable and can well alleviate perturbations of various models. In addition, as detailed in search robustness experiment (see Appendix C.1 [Lines 575-588]), our allocation functions searched with different initial seeds share the similar performance and similar expressions. (c) Based on (a) and (b), various LLMs share the common sparse allocation law and our search space and operations are architecture-agnostic, stable and robust to different search trials and models, resulting in good generality of our search allocation function. To further validate this, we re-search the allocation function in OPT-6.7B model with different size and architecture. Our directly transferred allocation function has quite similar formulation and performance with the re-search function, confirming its generality across LLMs to some extent. We will involve this discussion in the revision.

Table: Mean accuracies (%) of transferred and re-searched allocation functions with SparseGPT (Uniform baseline 55.19) for OPT-6.7B at 0.5 sparsity

Dear Reviewer gYq,

Thanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would appreciate it if the reviewer considers raising the score. Please see our responses below one by one:

---

Q1: Compare to OWL and evolution method.

A1: Our method's novelty lies in being the first automated search for adaptive sparsity allocation, significantly differing from traditional adaptive pruning methods like OWL [14] and evolution methods like Pruner-Zero [37]:

(1) Compare to OWL:

(a) Our method revolutionizes the field with an automated search approach that removes the need for expert design. In contrast, OWL relies on hand-designed, hyperparameter tuning and is limited by fixed form constraints.

(b) Our comprehensive search space systematically maps element-wise scores to sparsity ratios, whereas OWL only computes the keep ratio based on outlier ratio linearly, lacking our introduced nonlinear mapping operators.

(c) Comparative experiments show our method's significant outperformance over OWL in Table 8.

(2) Compare to evolution method: Our DSA differs Pruner-Zero [37] in method type, search space, task, strategy, and Input-Output characteristics (See Table below).

(a) We uniquely frame LLM sparsity allocation as an AutoML challenge, opening novel avenues for enhancing LLM efficiency.

(b) Our search space is customized for LLM sparsity allocation, integrating various operations in innovative ways.

(c) We develop LLM-specific acceleration techniques like program checking, making our approach practical for large-scale LLM optimization.

Please note that the title and abstract of Pruner-Zero [37] are accessible in May 2024, but its full paper is released (on arXiv and OpenReview) in June 2024, after the NeurIPS deadline. We already discuss OWL and Pruner-Zero in the related work [lines 143-148] and will add more comparisons in the revision.

Q2: More results for high sparsity ratios.

A2: Our responses are:

(1) To ensure fair comparisons, we follow standard settings like 50% sparsity, where baseline results are strong, making additional gains challenging.

(2) Our experiments in Table 8 at higher sparsity levels (65%-80%) on LLaMA-1 show our method's ability to achieve substantial improvements.

(3) Following the suggestions, we provide 60% and 70% sparse experiments in the rebuttal PDF (See our general response). Our method demonstrates gains ranging from 1.24% to 7.03% in LLaMA-2-7B&13B, and LLaMA-3-70B at 60% sparsity. At 70% sparsity, our improvements ranging from 1.91% to 7.68% underscore the effectiveness of our DSA for high sparsity ratios.

Q3: About typos.

A3: We appreciate the reviewer's corrections of typos in Erdos-Renyi and LLaMA-1-7B and commit to fixing them in the revision.

Q4: About potential combination with Pruner-Zero.

A4: Yes. Following the suggestion, we conduct experiments combining our method with Pruner-Zero, yielding new state-of-the-art results. This successful integration is due to the orthogonal nature of the two methods: Pruner-Zero optimizes element-wise importance scoring, while our DSA specializes in adaptive layer-wise sparsity allocation.

Table: Mean accuracies (%) at 0.5 sparsity on 7 zero-shot task.

Q4: About integration with 2:4 constraint.

A4: Following the suggestions, we evaluate our DSA under 2:4 sparsity on LLaVA (See Table below) and LLaMA-2 (See our general response). These consistent gains (2.2% ~ 2.8% on LLaVA and 1.4% ~ 4.2% on LLaMA-2) across different model sizes and datasets suggest that our approach is more effective at maintaining performance under tighter sparsity constraints.

---

Q5: Details of the time cost .

A5: As detailed in the experiment [Line 252-258], the 0.5-day search time mentioned is for the LLaMA-1 7B model on WikiText2 based on Wanda at 50% sparse setting. We will clarify this detail in the revision.

---

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.

Dear Reviewer gYqu,

We are profoundly grateful for your thorough and constructive comments. We have addressed your concerns point-by-point in our rebuttal and welcome any further inquiries. If our response has successfully alleviated your concerns and highlighted the merit of our work, we would be deeply appreciative if you could reconsider your recommendation. We promise to thoroughly reflect all your comments in the final manuscript. We have conscientiously incorporated feedback from all reviewers and hope this is positively reflected in your assessment. Thank you for generously dedicating your time to review our response.

With sincere gratitude,

Paper 66 Authors

Dear Reviewer gYqu,

Thank you again for spending your valuable time reviewing our paper. We have carefully considered and addressed your concerns. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal. We appreciate your constructive and insightful comments and will definitely incorporate them into the final manuscript.

Best regards,

Paper 66 Authors

Thanks for the authors' replies. Most of my questions have been addressed.

One remaining question is about the running time. As the author replied to Reviewer s93u: For other different LLM models, we directly transfer this discovered allocation function without additional search.

I'm wondering how this is realized in practice. Given a different model, for instance, the LLaMA2-70b or OPT model family, how can it be transferred into the new model which has different number of layers or model structure?

Dear Reviewer gYq,

We would like to express our sincere gratitude for your time and valuable feedback on our paper. In our rebuttal, we have carefully considered and incorporated your suggestions. We are eagerly anticipating any additional feedback you may have. Additionally, to further address your concerns regarding generalizability of allocation function in the running time, we make best efforts to clarify comprehensively as follows:

(1) Practical implementations of searched allocation function: The searched allocation function comprises four operations: LOG_ABSLOG (pre-process), GEOMETRIC_MEAN (reduction), COS (transformation), and EXP (post-process). (a) Among them, our pre-process operation is element-wise and architecture-agnostic, similar to metrics in SparseGPT or Wanda. Our reduction operation, akin to OWL's outlier operator, processes intra-layer scores to importances, independent from layer lengths and generalize to various model structures. (b) As alternative to OWL linear expression, our transformation post-process operations are non-parametric functions and convert per-layer importances → sparsity ratios. They (implemented as "COS" and "EXP" using PyTorch's torch.cos and torch.exp functions) can handle inputs of different lengths (corresponding to different layer counts), making them adaptable to diverse model sizes. (c) In summary, our searched allocation function can be applied like OWL to different numbers of layers or model structures. We have also provided the implementation codes in the Supplementary Material and will open-source them upon acceptance. The core practical code of our searched allocation function is in the Supplementary Material's code\lib\autolayer.py file.

(2) Understanding generalizability: Sufficient results in Table 2 &3 &4 & 5 [Line 250-286] and rebuttal PDF (See our general response) solidly demonstrates the generality of our allocation functions across different models and tasks. We provide several aspects to understand its generality: (a) Different LLMs share the common sparsity laws (e.g., important weights have salient gradients) resulting in sparsity methods with fixed metrics that can be generalized to various models. For example, Wanda and OWL apply the same metric for different tasks without discussing whether their functions need to be changed for different structure and depth models. Our allocation function search space is based on observation element-wise score distributions in LLMs, as discussed in the Introduction [Lines 62-78] and Figure 1(left). This observation aligns with OWL and the sparsity laws that the initial layers of LLMs are more important. Thus, our searched allocation function naturally follows the sparsity laws, allowing good generalizability to the different LLMs. (b) The operators contained in our allocation function are architecture-agnostic and serve to normalize and eliminate architecture variance. For example, our pre-process operator standardizes inputs by normalizing scores across layers, ensuring consistent performance metrics by addressing scale variations. As our stability analyses [Lines 243-250] and theoretical understanding in Appendix B [Lines 554-571], our allocation function is stable and can well alleviate perturbations of various models. As detailed in search robustness experiment (see Appendix C.1 [Lines 575-588]), our allocation functions searched with different initial seeds share similar performance and expressions. (c) Based on (a) and (b), various LLMs share the common sparse allocation law and our search space and operations are architecture-agnostic, stable and robust to different search trials and models, resulting in good generality of our search allocation function. To further validate this, we re-search the allocation function in OPT-6.7B model with different size and architecture. Our directly transferred allocation function has quite similar formulation and performance with the re-search function, confirming its generality across LLMs to some extent. We will involve this discussion in the revision.

Table: Mean accuracies (%) of transferred and re-searched allocation functions with SparseGPT (Uniform baseline 55.19) for OPT-6.7B at 0.5 sparsity

---

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We promise to include these discussions in the revision and sincerely hope that reviewer will consider improving the recommendation.

Best regards,

Paper 66 Authors

Dear Reviewer s93u,

Thanks for constructive comments. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would appreciate it if the reviewer considers raising the score. Please see our responses below one by one：

---

Q1: About results on structured sparsity.

A1: Our responses are: (1) Following the suggestion, we apply our non-uniform layer-wise sparsity allocation method to the structured pruning technique LLM Pruner [32], which accelerates pruned LLMs directly. The results in Table below show that our method performs well in structured pruning scenarios and outperforms OWL.

Table: Perplexity of Structure Pruning with LLaMA-7B on WikiText-2.

(2) We would like to highlight that our method supports and enhances 2:4 and 4:8 structured sparsity, compatible with Nvidia GPUs for hardware acceleration. The results presented in Tables 6 and 7, as well as the results in the rebuttal PDF (See our general response), demonstrate the efficacy of our approach, showcasing gains of 1.4% ~ 4.2% at 2:4 sparsity and ~1% gains at 4:8 sparsity. These gains are promising advancements in this area (Recognized by Reviewer 1f5c, of38, gYqu, and VeXf).

(3) Recent advancements in advanced GPU kernels like NVIDIA cuSPARSE [ref1], Sputnik [ref2], and Flash-LLM [ref3] have rapidly supported unstructured sparsity. This practical relevance extends beyond GPUs to non-GPU hardware such as CPUs (e.g., XNNPACK [ref4]) and specialized accelerators like FPGA accelerator. Our method under unstructured sparsity also achieves 1.8x~3.7x speed up with DeepSparse inference engine for LLaMA-V2-7B-chat-hf, as presented in Table below.

---

Q2: Details of the time cost .

A2: As detailed in the experiment [Line 252-258], the 0.5-day search time mentioned is for the LLaMA-1 7B model on WikiText2 based on Wanda at 50% sparse setting. For other different LLM models, we directly transfer this discovered allocation function without additional search. We apologize for not explicitly stating this and will clarify this detail in the revision.

References:

[ref1] NVIDIA GPUs scalability to solve multiple (batch) tridiagonal systems implementation of cuThomasBatch. In PPAM 2018.

[ref2] Sparse gpu kernels for deep learning. In SC conference 2020.

[ref3] Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. ArXiv:2309.10285.

[ref4] Fast sparse convnets. In CVPR2020.

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.

Dear Reviewer s93u,

We are deeply thankful for your careful consideration and constructive feedback. Our rebuttal addresses your concerns in detail, and we are open to any additional questions. If our response has successfully clarified the value of our work and addressed your concerns, we would be incredibly grateful if you could reconsider your recommendation. We promise to thoroughly reflect all your comments in the final manuscript. We have diligently incorporated feedback from all reviewers and hope this effort is evident in your evaluation. Thank you for your time and expertise in reviewing our response.

With utmost respect,

Paper 66 Authors

Dear Reviewer s93u,

Thank you for your time reviewing the paper. Your constructive feedback will help improve the quality of our paper.

We have also addressed all your concerns in our rebuttal point-by-point. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be truly grateful if you could reconsider your recommendation.

We thank you again for your time!

Best regards,

Paper 66 Authors

Dear Reviewer s93u,

We greatly appreciate the time and effort you've invested in reviewing our paper. Your constructive feedback has been invaluable in enhancing the quality of our work.

We've diligently addressed all your concerns in our point-by-point rebuttal. As we approach the conclusion of the author-reviewer discussion period, we kindly request that you review our responses. Other two reviewers have raised their scores after considering our rebuttals and revisions. In light of this, we sincerely hope you'll also reconsider your recommendation and potentially improve your score.

Once again, we extend our heartfelt thanks for your time and expertise. Your insights have been crucial to the refinement of our paper.

Best regards,

Paper 66 Authors

Dear Reviewer of38,

Thanks for constructive comments. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would appreciate it if the reviewer considers raising the score. Please see our responses below one by one：

---

Q1: More results on higher sparsity levels.

A1: Our responses are:

(1) To ensure fair comparisons, we follow standard settings like 50% sparsity, where baseline results are strong, making additional gains challenging.

(2) Our experiments in Table 8 at higher sparsity levels (65%-80%) on LLaMA-1 show our method's ability to achieve substantial improvements (Recognized by Reviewer s93u, gYqu, and VeXf).

(3) Following the suggestions, we provide 60% and 70% sparse experiments in the rebuttal PDF (See our general response). Our method demonstrates gains ranging from 1.24% to 7.03% at 60% sparsity and 1.91% to 7.68% at 70% sparsity, which underscore our effectiveness for high sparsity ratios.

---

Q2: About motivation and insights of design of the search space.

A2: Our responses are:

(1) As discussed in Introduction [Lines 62-78] and Figure 1 (left), our design is motivated by analyzing element-wise score distributions, and mean values of per-layer scores, inspiring reduction and transformation ops. While basic reduction showed modest gains, applying transform ops yielded more promising results. We also include pre-processing for score normalization and post-processing to enhance function fit.

(2) Converting element-wise scores to layer-wise scores is important: layer-wise scores provide a consolidated view of importance, reducing noise and offering more stable ratios. By focusing on layer-wise metrics, critical layer information can be leveraged for better parameter retention. This process enhances computational efficiency by simplifying complexity from the element to the layer level.

(3) As detailed in Primary Operators section [Lines 184-197], key insights of our design are:

(a) Pre-process: Standardizes inputs by normalizing scores across layers, ensuring consistent performance metrics by addressing scale variations.

(b) Reduction: Condenses element-wise information by extracting representative values per layer, reducing computational complexity.

(c) Transformation: Models complex relationships with functions, enabling the representation of intricate patterns in layer importance.

(d) Post-process: Fine-tunes the allocation function for optimization, enhancing flexibility.

(4) Ablation study (See Table below) shows that the Reduction ops is the most influential, followed by the Transformation ops.

Table : Perplexity with LLaMA 7B at 0.7 sparsity on WikiText2

Q3: About practical runtime speedup.

A3: Following the suggestions, we evaluate the runtime speedup (See Table below) of employing DeepSparse inference engine on LLaMA-V2-7B-chat-hf model. The findings indicate that our method exhibits comparable speedup to uniform pruning.

Q4: About generalization of allocation functions.

A4: Yes. We directly transfer top-performing allocation function in Equation 6 to different models and takes without additional search. Our experiments show this allocation functions do show good generalization across models and sparsity settings.

Q5: About applicability to non-Transformer networks.

A5: We would like to highlight that our method is architecture-agnostic and can be applied to various model types, including CNNs. The reason is that our search space is comprehensive and includes diverse types of operations that can be utilized across different model architectures. To confirm this, we apply our allocation functions to ConvNeXt in Table below. Our DSA surpasses other methods, especially at higher sparsity levels, showing its effectiveness with CNNs.

Table: Accuracy (%) of Sparse ConvNeXt-Base on ImageNet-1K.

Q6: About experiment with on LLaMA-3 70B.

A6: Following the suggestion, we perform experiments on LLaMA-3 70B in the rebuttal PDF (See our general response). Our method shows 2.23% to 4.38% gains across different pruning approaches, aligning with our effectiveness on multiple LLMs.

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.

Dear Reviewer of38,

We extend our heartfelt thanks for your meticulous and invaluable comments. Our rebuttal addresses your concerns comprehensively, and we welcome any further inquiries you may have. If our response alleviates your concerns and clarifies the value of our paper, we would be truly grateful if you could reconsider your recommendation. We pledge to thoroughly incorporate all your astute observations in the final version. We have conscientiously integrated feedback from all reviewers and hope this is reflected favorably in your assessment. We are truly grateful for the time you've invested in reviewing our work.

With sincere appreciation,

Paper 66 Authors

Dear Reviewer of38,

Thank you for your time reviewing the paper. Your constructive feedback will help improve the quality of our paper.

We have also addressed all your concerns in our rebuttal point-by-point. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be truly grateful if you could reconsider your recommendation.

We thank you again for your time!

Best regards,

Paper 66 Authors

Dear Reviewer 1f5c,

Thanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would appreciate it if the reviewer considers raising the score. Please see our responses below one by one：

---

Q1: About motivation and insights of allocation function design.

A1: Our responses are:

(1) Our allocation function design is motivated by analyzing element-wise score distributions, as discussed in the Introduction [Lines 62-78] and Figure 1(left).

(a) We notice that mean, variance, and entropy values of per-layer element-wise scores can serve as allocation indicators, inspiring reduction operations.

(b) While basic reduction of element-wise scores showed modest improvements, applying transform functions yielded more promising results, prompting the introduction of transform operations.

(c) We include pre-process to normalize scores for fair comparison and post-process to further enhance function fit's upper bound.

(2) The key insight of our four-component design is its flexibility in exploring diverse allocation functions tailored to each LLM's characteristics, while capturing complex, non-linear relationships between element-wise scores and sparsity ratios. As detailed in the Primary Operators section [Lines 184-197], each component serves a specific purpose:

(a) Pre-process: Standardizes inputs by normalizing scores across layers, ensuring consistent performance metrics by addressing scale variations.

(b) Reduction: Condenses element-wise information by extracting representative values per layer through operations like variance or entropy, reducing computational complexity.

(c) Transformation: Models complex relationships with functions like sine or exponential, enabling the representation of intricate patterns in layer importance.

(d) Post-process: Fine-tunes the allocation function for optimization, enhancing flexibility.

(3) Ablation study (See Table below) shows that the Reduction component is the most influential, followed by the Transformation component, while the Pre-processing and Post-processing components have a smaller impact on the overall DSA performance.

Table : Perplexity with LLaMA 7B at 0.7sparsity on WikiText2

---

Q2: About performance improvements.

A1: Our responses are:

(1) We would like to clarify that our method consistently demonstrates substantial enhancements (1%-14%) across a spectrum of scales (7B~70B), models (LLaMA-1|2|3, Mistral, LLaVA, and OPT models), and complex tasks (reasoning and multimodal benchmarks), which robustly confirms its effectiveness. The magnitude of gains naturally varies depending on the model and task. For instance, in Tables 6 & 7, considering the Dense LLaVA-1.5 Models ranging from 7B to 13B, we observe a only 1.5% performance increase on VQAv2. Notably, our around 1% gains over Wanda, the current state-of-the-art pruning method, is promising advancements in this area (Recognized by Reviewer of38, s93u, gYqu, and VeXf).

(2) To ensure fair comparisons, we follow standard settings like 50% sparsity, where baseline results are strong, making additional gains challenging.

(3) Our experiments in Table 8 at higher sparsity levels (65% ~ 80%) on LLaMA-1 show our method's ability to achieve substantial improvements.

(4) Additional results in the rebuttal PDF (See our general response) further demonstrate that our method can achieve substantial gains, with improvements ranging from 1% ~ 7% at 60% sparsity, 2% ~ 7.6% at 70% sparsity, and 1.4% ~ 4.2% at 2:4 sparsity.

Q3: About performance in 2:4 sparsity setting.

A3: Following the suggestions, we evaluate our DSA under 2:4 sparsity on LLaVA (See Table below) and LLaMA-2 (See our general response). These consistent gains (2.2% ~ 2.8% on LLaVA and 1.4% ~ 4.2% on LLaMA-2) across different model sizes and datasets suggest that our approach is more effective at maintaining performance under tighter sparsity constraints.

Finally, we hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.

Dear Reviewer 1f5c,

We sincerely appreciate your thoughtful and constructive feedback. We have diligently addressed each of your concerns in our point-by-point rebuttal. We hope our response has alleviated your concerns and illuminated the value of our work. If so, we would be immensely grateful if you could reconsider your recommendation. We assure you that all your insightful comments will be meticulously incorporated into the final manuscript. We have earnestly integrated feedback from all four reviewers and hope this is evident in your evaluation. Thank you for dedicating your valuable time to review our response.

With deepest gratitude,

Paper 66 Authors

Dear Reviewer 1f5c,

We sincerely thank you for your valuable feedback, which has provided our paper with deeper insights. We promise to thoroughly reflect all your comments in the final manuscript. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be immensely grateful if you could reconsider your recommendation.

Best regards,

Paper 66 Authors