We thank you for your feedback and positive assessment of our work. Below, we address your specific questions in detail.

Impact of Introducing Additional Zeros in N:M Pattern

Thank you for this question. As demonstrated in Theorem 2.2 (Page 4), by varying the backward pass mask across iterations, the expected value of the gradients matches the original unpruned gradient. This ensures that introducing additional zeros does not significantly degrade accuracy. To provide an intuitive explanation, in each backward pass, a subset of weights is pruned. By dynamically changing the location of these zeros across iterations, the SLoPe input gradients converged toward the original input gradients over time, maintaining accuracy.

Memory Footprint Reduction During Pretraining and Inference

You raise an important point regarding memory footprint reduction. During pretraining, we store both the weights and their transposed copies to enable the double-pruned backward pass. This introduces additional memory overhead, narrowing the gap between the end-to-end memory reduction for pretraining (68%) and inference (54%).

Below, we provide a detailed analysis of memory savings per four consecutive elements, which can be generalized to larger model sizes. This analysis has been included in Revised-Paper-Page-7.

Inference Memory Reduction:

Dense Model:

Weight: 4 * 16 bits

Sparse Model:

Weight: 2 * 16 bits (Non-zeros) + 3 bits (Indices)

Memory Saving (2:4 Sparse vs. Dense)

(2 * 16 + 3) / (4 * 16) = 54%

Training Memory Reduction:

Dense Model:

Weight: 4 * 16

Gradient: 4 * 16

Optimizer States: 2 * 4 * 32

Sparse Model with 2:4 Pattern:

Weight: 2 * 16 bits (Non-zeros) + 3 bits (Indices)

Weight-Transpose: 2 * 16 bits (Non-zeros) + 3 bits (Indices)

Binary Mask: 4 * 8 bits

Gradient: 2 * 16 bits 

Optimizer States: 2 * 2 * 32

Memory Saving (2:4 Sparse vs. Dense)

(2 * 16 + 3 + 2 * 16 + 3 + 4 * 8 + 2 * 16 + 2 * 2 * 32) / (4 * 16 + 4 * 16 + 2 * 4 * 32) = 68%

Choice of Low-Rank Adapter Sizes

Thank you for your observation regarding the size of low-rank adapters. To maximize the speedups offered by SLoPe during inference, it is essential to minimize the overhead introduced by low-rank adapters. This consideration led us to select relatively small low-rank adapter sizes compared to the model’s hidden dimensions. This choice ensures that the computational overhead remains minimal while maintaining performance benefits.

We hope these responses address your concerns and provide additional clarity. Thank you again for your valuable feedback.

Part 1 / 2

We thank the reviewer for their detailed feedback and for pointing out areas where additional clarification and evaluation are needed. Below, we address each of your questions and concerns.

Evaluation of Sparsity Impact and Flexibility

Thank you for highlighting the importance of a broader evaluation. To address this, we have included results on additional zero-shot downstream teaks using the GLUE benchmark, consistent with the FST paper. These evaluations were conducted using the Language Modeling Harness and the results are summarized in the table below. We are also in contact with the authors of the FST paper to obtain their fine-tuning scripts and will incorporate fine-tuning results as soon as they are available.

Our results demonstrate that SLoPe outperforms Extended SR-STE across various tasks on average, achieving a balance between speed and performance while avoiding the drawbacks of dynamic mask updates. We hope these additional results strengthen the evidence for the effectiveness of our approach across a broader range of tasks.

Regarding the use of static sparsity, we understand the concern about flexibility. However, dynamic sparsity methods, as demonstrated in SR-STE [1], can introduce instability due to frequent mask updates, potentially impacting training. We have elaborated on these trade-offs in Paper-Appendix-A-Page-15.

Clarification on Baselines

We apologize for any confusion regarding the terminology. SR-STE is limited to SGD optimizer. The FST paper extends SR-STE for optimizers other than SGD, and we are using that extension in our experiments. Therefore, we have used these terms interchangeably in our accuracy experiments. Additionally, FST provides a fast implementation of the extended optimization method in PyTorch, which we have compared against in our speedup and memory reduction experiments.

Comparison with Smaller Dense Models

In response to your suggestion, we have added zero-shot GLUE results and trained a dense GPT model with half the layers of GPT-2 Small (GPT-2 Half) to compare performance with sparse models at equivalent parameter count. The table below summarizes the results, showing that SLoPe, especially with low-rank adapters, consistently outperforms both GPT-2 Half and Extended SR-STE across tasks.

LoRA Pretraining Impact and Ablations

Thank you for your observations regarding the impact of low-rank adapters (LoRA).

LoRA impact during Pre-training $\rightarrow$ Low-Rank adapters in SLoPe improve accuracy by up to 4.8% on GLUE tasks (CoLA in Table 10 in Page 22) and 0.4% on SQuAD for BERT-Large-Uncased.

LoRA impact on SR-STE $\rightarrow$ In addition, our proposed lazy low-rank adapters can boost the accuracy of prior work, such as SR-STE, by up to 1.6% on the OpenBookQA downstream task for GPT-2 Small (Table 3 in Page 9).

LoRA Ablations $\rightarrow$ We had the LoRA ablations for MMLU, Arch Challenge, and OpenBookQA in Table 3 (Page 9) of the submitted paper. The Adapter Rank refers to the relative rank of the LoRA compared to the hidden dimension of the model.

Clarification on Inference Speedup and Sparse Pre-Training

Thank you for your insightful questions.

FST uses a dense fine-tuning step (as an accuracy preserving mechanism), compromising ~17% of the pretraining time (see section 4.4 in FST [2]), resulting in a dense model for inference with no inference speedup. In contrast, SLoPe uses lightweight low-rank adapters in the last 1% of pretraining as an accuracy preserving mechanism, maintaining sparse computation benefits. The goal of FST is exclusively to accelerate the training of LLMs, whereas SLoPe aims to accelerate both training and inference.

SLoPe inference computation $\rightarrow$ During inference, SLoPe does not merge low-rank adapters with the weights. Instead, it employs an efficient sparse matrix-matrix multiplication alongside low-rank adapters, yielding faster inference than dense models. The inference speedups reported for SLoPe were measured using this implementation, whereas the dense inference time for FST reflects its dense fine-tuned model.

SLoPe downstream finetuning $\rightarrow$ During downstream fine-tuning, SLoPe retains the sparsity mask, updating only the non-zero weights and low-rank adapters, which leads to inference acceleration even after finetuning.

SLoPe Speedup Measurements $\rightarrow$ The speedups in the SLoPe column are measured by timing this efficient sparse+low-rank implementation and comparing it with a dense model. For FST, the model at inference is dense because of using dense fine-tuning as their accuracy preserving mechanism, resulting in zero speedup at inference time.

Clarification on Flash Attention Usage

Thank you for pointing this out. We confirm that Flash Attention 2 was enabled across all baselines, including dense models, FST, and SLoPe. We will clarify this in the revised manuscript to avoid confusion.

[1] Hu et al, “S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training,” NeurIPS 2024

[2] Hu et al, “Accelerating Transformer Pre-training with 2:4 Sparsity,” ICML 2024

I sincerely appreciate the authors' efforts to address many of my concerns. However, after carefully reviewing the authors' responses to my comments, as well as those provided to other reviewers, I still have some outstanding concerns:

1. Could the authors confirm that all reported SR-STE values in this paper are indeed for FST? What optimizer was used in the experiments? Specifically, does this imply that every contribution and improvement of FST has been accounted for, and the results presented reflect the best possible outcomes for FST? I emphasize this point because there is not a single experiment that is directly comparable between this paper and FST. Either the models, the tasks, or the settings (e.g., zero-shot versus fine-tuning) differ, making it impossible to directly compare this work with what the authors refer to as the latest advancements in sparse pre-training. Moreover, the authors mention that the fine-tuning code of FST is not provided. However, elsewhere, they note that FST performs dense fine-tuning, implying that implementing FST fine-tuning should be straightforward.

2. Regarding the smaller dense model (Half), it should be designed by reducing the hidden dimensions of each layer, rather than by removing entire blocks. This approach is what was done in FST. Depth pruning generally has a more detrimental effect on model performance compared to width pruning (or reducing the hidden dimensions). For example, this can be observed in Figure 5 of [1]. Besides, the pattern of sparsity introduced in SLoPE resembles width-pruning.

3. The zero-shot accuracies on GLUE are particularly concerning, especially for SLoPE with r=2.1% and baseline SR-STE. These results further highlight the significant impact of a static sparsity mask on performance compared to a dynamic one. While I understand the inherent trade-offs between flexibility, speedup, and fine-tuning accuracy—and that SLoPE aims to balance both pre-training and fine-tuning speed—it is crucial for readers to see accurate results for the smaller dense model. This is necessary to enable a clear assessment of the trade-offs offered by SLoPE.

[1] LLM Pruning and Distillation in Practice: The Minitron Approach, Sreenivas et. al, Arxiv 2408.11796

Depth-Pruning vs. Width-Pruning for Reduced-Sized Dense Models

Thank you for bringing this paper to our attention. After carefully reviewing the paper, we would like to clarify that [1] focuses on comparing the quality of depth-pruning vs. width-pruning after full dense pretraining. Specifically, the authors pretrain a dense model of size P and then prune it during finetuning to obtain a reduced-sized model of approximately P/2 for downstream tasks. the model for the downstream tasks to obtain a reduced-sized model of size ~P/2. We will include a detailed discussion to highlight the differences between their method and ours, ensuring that the contributions of both works are properly contextualized.

While the context differs from our focus on sparse pretraining methods, we found your suggestion intriguing and initiated a set of experiments to explore the tradeoffs between these pruning strategies in our setting. We look forward to incorporating these insights into our work and thank you again for the thoughtful suggestion.

Disclaimer: We pretrained LLaMA-2-7b and Gemma2-2b and Gemma2-9b models using the MaxText codebase on TPU. This choice was motivated by access to a larger pool of TPU resources, which enabled us to obtain preliminary results more quickly and share them within the discussion period. Please see the configurations for each model in Appendix-Table-15,16,17-Page-26,27.

Results: Preliminary pretraining loss curves suggest no significance difference between depth-pruning and width-pruning during pretraining. Interestingly, in some cases, depth-pruning appears to outperform width-pruning. These early loss curves are included in the Appendix-Figure-10-Page-28.

We acknowledge that these are early results, shared to provide insights before the discussion period concludes. Thanks to your comment, we believe that a comprehensive analysis of the impact of depth-pruning vs. width-pruning represents a valuable avenue for future work.

Zero-shot Accuracies on GLUE

Thank you for reviewing the results. We understand that per-task model quality could differ from one method to another. A couple of clarifications for comparing model quality between models with the same number of parameters:

(1) Zero-Shot GLUE Results: SLoPe outperforms Extended SR-STE in 3/8 tasks as well as average (SLoPE = 43.18 vs. Extended SR-STE = 42.6).

(2) Modern Zero-Shot Benchmarks: In more recent zero-shot eval benchmarks (MMLU, Arch Challenge, and Open Book QA) Slope outperforms Extended SR-STE in 2/3 tasks as well as the average quality (19.57 vs. 18.90)

Additional Zero-Shot Results

To further demonstrate the effectiveness of SLoPe, we conducted additional evaluations on five modern zero-shot benchmark tasks using the Language Model Evaluation Harness.

As shown in the table below, SLoPe consistently outperforms Extended SR-STE (same number of parameters) on four out of five tasks. These results underscore the advantages of our method’s static sparsity masks. This performance gain highlights the robustness and practicality of SLoPe in leveraging sparsity during pretraining across diverse tasks.

Additional results, including evaluations on different configurations and metrics, are provided in the (Section-3.2-Table-3, Page-9) for further validation.

References

[1] Sreenivas et. al, “LLM Pruning and Distillation in Practice: The Minitron Approach,” Arxiv 2024

Notations

To clarify our response, we have included a set of notations used throughout. This table is also available in the revised manuscript Appendix-R-Table-14-Page-24.

Clarification about SR-STE and FST [1]

We apologize for the confusion and appreciate your comment. To address this, we have revised the manuscript to clearly specify which algorithm (Extended SR-STE or FST) is being used in each comparison. Below, we provide a detailed explanation of these algorithms and the rationale for our choices in the comparisons:

FST → We compare SLoPe with FST exclusively for training speedups and memory savings (Table-1-Page-7 and Table-2-Page-8). Comparing pretraining quality between SLoPE and FST is less meaningful because the final models produced by these methods differ significantly in the number of parameters (See Table below). Specifically, FST produces a dense model after sparse pretraining and dense finetuning (83% sparse pretraining + 17% dense finetuning), while SLoPe produces a sparse model augmented with lightweight low-rank adaptors (99% sparse pretraining + 1% low-rank adaptation). The final number of parameters after pretraining in SLoPe is approximately 1.48$\times$ smaller than in the Dense (or FST) model, which makes a direct quality comparison imbalance. That said, we welcome your feedback on how to conduct a meaningful head-to-head quality comparison with FST.

Extended SR-STE → We compare SLoPe with Extended SR-STE in terms of model quality, focusing on understanding the dynamics between static and dynamic masking under an equal number of parameters. This allows for a fair, "apple-to-apple" comparison between the methods (iso-params). We refer to this method as "Extended SR-STE" because, while the original SR-STE approach was designed for use with SGD, the FST paper extended it to support other optimizers.

Choice of Optimizer

We use ADAMW for both SLoPE and Extended SR-STE. For Extended SR-STE, we strictly follow the mechanism proposed in the FST paper, including the application of masked decay on the gradients. We believe this approach ensures a fair and comprehensive integration of all relevant contributions from the FST paper into our quality comparisons. As mentioned earlier, FST is used exclusively for speedup comparisons, not quality comparisons, to highlight its intended strengths in those aspects. For ease of reference, we added code snippets of our implementation and FST implementation to Appendix-R-Page-24-25.

Fine-tuning Code of FST

Thank you for your suggestion. The publicly available FST codebase includes implementations for sparse pretraining and dense finetuning (this is extended pretraining, please see the notation table above), which we have adopted in our comparisons. However, the authors have not released the codebase nor the configurations for downstream finetuning on tasks like GLUE, making it challenging for us to reproduce their results in Table 6 at this time. The authors have assured us that they will share the downstream finetuning code after their deadline, and we plan to run the relevant experiments as soon as we gain access to it. Until then, all results we present are based on zero-shot downstream evaluation, without any additional downstream finetuning.

References

[1] Hu et al, “Accelerating Transformer Pre-training with 2:4 Sparsity,” ICML 2024

I thank the authors for their detailed response and the clarifications provided.

Apples and Oranges

I did not receive a definitive answer to one of my key questions. While the authors argue that comparing the accuracies of SLoPE and FST constitutes an "apples-to-oranges" comparison, I find it inconsistent that much of Table 1 and Table 2 is devoted to comparing the speed-up and memory usage of SLoPE with FST, using values from the dense model resulting from the application of FST. Wouldn’t this also fall under the category of an "apples-to-oranges" comparison?

I would like to clarify my question. I apologize if my original question was not clear:
Are the results reported as "Extended SR-STE" identical to those of FST without the full fine-tuning stage, assuming an independent individual were to reproduce the results presented in this paper?

Dense Baseline

I find it perplexing that significant resources were allocated to these additional width-depth sparsity comparison experiments, yet no dense pretraining experiment was conducted using the correct dense baseline. This seems particularly crucial given the role of a dense baseline in contextualizing the sparsity trade-offs of SLoPE. As I said before, the sparsity pattern introduced in SLoPE resembles width-pruning.

While I understand that the discussion period has now concluded and no further experiments can be added, my original point still stands: this paper lacks a proper dense baseline designed by reducing the width of each layer rather than removing entire blocks. This omission limits the ability to fully assess the advantages and trade-offs of SLoPE compared to an appropriate dense baseline.

Dear Reviewer NKfR,

Thank you for your active engagement during the rebuttal period and your suggestions.

Apples and Oranges

We compare SLoPe with FST for speedup and memory savings as FST is the only baseline in sparse pretraining that demonstrates practical acceleration and memory savings on GPUs.

To provide further context, the table below presents a detailed comparison of SLoPe and Extended-SR-STE in terms of speedup and memory savings:

< Speedup ($\uparrow$ is better) >

< Memory Reduction ($\downarrow$ is better) >

$^*$ Values larger than 1.00 in the memory reduction table indicate memory overhead. Extended-SR-STE (and FST during their sparse pretraining) stores the dense weight, in addition to the sparse weight and its transpose, leading to a memory overhead in comparison to the dense baseline.

$^*$ The inference speedup and memory reduction of Extended-SR-STE and SLoPe are identical, when using the efficient SLoPe tiling and low-rank fusion implementation.

Identical Extended SR-STE

Are the results reported as "Extended SR-STE" identical to those of FST without the full fine-tuning stage, assuming an independent individual were to reproduce the results presented in this paper?

Yes.

We apologize if our answer was not clear enough in our previous response. As we mentioned:

• [Part 1/3] Notation Table clearly states that Extended-SR-STE is the sparse pretraining part of the FST algorithm.

• [Part 1/3] In the Choice of Optimizer section we mentioned that we provided the code snippet for FST and Extended SR-STE (our implementation) that shows we use the same implementation.

Dense Baseline

We want to reassure you that we’re doing our best to address all comments thoroughly and on time. The additional experiments during the discussion period were possible because we had access to dedicated TPU resources, which allowed us to provide results quickly. Unfortunately, the GPT experiments are taking longer since they’re running on a shared GPU cluster (64×V100). We’re working hard to get these results ready by the deadline and appreciate the reviewer’s understanding of these practical limitations.

On the comparison between depth-pruning and width-pruning, we were intrigued by your comment. In our experiment across three models, we observed that depth-pruning and width-pruning consistently resulted in identical loss values during pre-training. This was true across different model architectures, which makes us believe the two approaches lead to equivalent outcomes in this context. If there’s something we might be overlooking or if you have reasons to expect different results, we’d love to hear your perspective and explore this further.

We thank you for your feedback and positive assessment of our work. Below, we address your specific questions in detail.

Novelty in Combining Sparsity and Low-Rank Adapters for Pretraining

We appreciate your recognition of the importance of sparsity and low-rank adapters in prior work, such as LoSparse [1]. While these techniques have indeed been studied previously, our work introduces a unique approach by applying sparsity and low-rank adapters in both the forward and backward passes (doubly pruned), accelerating not only model training but also inference. This algorithmic contribution is further complemented by the implementation of new tiling and low-rank inference CUDA kernels, enabling more efficient training and inference.

Additionally, LoSparse [1] uses low-rank adapters throughout the entire fine-tuning process, which limits the potential speedup and incurs additional computational overhead. In contrast, our approach strategically integrates lazy low-rank adapters exclusively during the final 1% of pretraining, effectively reducing overhead while maintaining their expressive capacity. This combination optimizes both efficiency and performance during pretraining, addressing a key gap in existing techniques.

We have already cited LoSparse in the paper, but in response to your feedback, we have included additional details in Appendix-Section-N (Page 23).

Clarification on Related Work and Baseline Coverage

We thank the reviewer for highlighting the need for a more detailed discussion of related work and baseline methods. To address this, we have expanded the related work section in Section N of the appendix (Page 22) to include a comprehensive discussion of baseline methods, such as FST, and other relevant studies. Additionally, we have highlighted key related works in the introduction. We hope this combination offers a balanced overview, but we are open to further expanding these sections to improve clarity for readers unfamiliar with the field.

Additional Zero-Shot GLUE Results

Thank you for the suggestion to include results on additional zero-shot datasets in GLUE. We have added an extended zero-shot evaluation using the GLUE benchmark with the Language Modeling Harness. The table below presents the performance of dense, SR-STE, and SLoPe training schemes, showing that SLoPe outperforms Extended SR-STE on average.

Additionally, we have contacted the authors of the FST paper to obtain their fine-tuning scripts and will incorporate fine-tuning GLUE results once available. We hope these additional results provide more comprehensive comparisons and address your concerns.

We have included this table in the revised manuscript, Appendix-Section-Q-Table-13 (Page 24).

Addressing Typos and Formatting Issues

Thank you for highlighting the typos and formatting inconsistencies, particularly the bold value errors in the tables. We apologize for these mistakes and have corrected them in the revised manuscript.

[1] Li, Yixiao, et al. "LoSparse: Structured compression of large language models based on low-rank and sparse approximation." ICML 2023

Dear Reviewer ywJx,

Thank you once again for your time and thoughtful feedback on our submission. As we approach the end of the discussion period, we wanted to gently remind you that we have updated the manuscript to address the questions raised. Specifically, we have revised the paper to clarify the notations and provided a fair comparison between FST and SLoPe.

Please refer to our three-part response to Reviewer NKfR (Part1, Part2, and Part3) for details.

Notations

To clarify our response, we have included a set of notations used throughout. This table is also available in the revised manuscript Appendix-R-Table-14-Page-24.

Eight Modern Zero-shot Benchmarks

We also added eight modern zero-shot benchmarks from Language Model Evaluation Harness. The tables below summarize the results. SLoPe outperforms Extended SR-STE (same number of parameters) on six out of eight tasks.

Initial zero-shot benchmarks

Additional Zero-shot Results

We would greatly appreciate your thoughts on the revisions, as your feedback is invaluable to ensuring the work is evaluated fairly and comprehensively. Thank you for your attention during this process.

Best regards,

The Authors

Thanks for the detailed reply from the authors. Most of my concerns have been addressed. I'd like to retain my score as positive.