Beyond 2:4: Exploring V:N:M Sparsity for Efficient Transformer Inference on GPUs

Q2: How did you implement the kernel using sparse tensor core? For higher sparsity (like 2:12, 2:24), the observed speedup falls short of the theoretical speedup, possibly due to inefficiencies in the implementation. Moreover, would you consider using cusparse/cublas or other CUDA kernels to compare speedup with the proposed method at highly sparsity?

A2: Thank you.

1. We utilize the same GPU kernel as employed in the Venom paper to accelerate V:N:M-sparse Transformers. For clarity, we have highlighted key distinctions between our work and the initial V:N:M work in the revised paper. First, as shown in Figure 2, our work and the initial work for V:N:M sparsity lies in distinct phases in the workflow of compressing a Transformer. The initial work primarily proposes GPU acceleration kernels, i.e., Figure 2(b). In contrast, our research emphasizes enhancing the accuracy of V:N:M-sparse Transformers under various scenarios. Our proposed framework within the three novel techniques is depicted in Figure 2(a).

   Besides, here we also would like to detail the significantly distinguishable contributions and advancements our work makes beyond the initial V:N:M sparsity study and other related works:

   • our work identifies and addresses critical issues that have long affected the field of Transformer 2:4 sparsity, yet have received insufficient attention until now. That is, 2:4-sparse Transformers in practice provide very limited end-to-end speedups, while the promising V:N:M sparsity, although capable of delivering higher speedups, has not demonstrated comparable accuracy for sparse Transformers relative to 2:4 sparsity. Our work exactly aims to bridge this gap by proposing a framework with three innovative techniques that algin the accuracy V:N:M sparsity with that of 2:4 sparsity. In doing so, our work significantly contributes to the field by promoting the widespread adoption of V:N:M sparsity as a truly effective solution for compressing Transformers.
   • Our work tackles three key fundamental challenges that prevented V:N:M-sparse Transformer from achieving the desired accuracy—challenges that have never been addressed in previous research. The three challenges are:(1) selecting appropriate values for V and M, (2) enhancing the accuracy of V:N:M sparsity under limited training budgets, and (3) implementing V:N:M-sparse LoRA training for large language models (LLMs). These challenges are critical and cannot be overlooked. Accordingly, our proposed three techniques to address challenges are also novel contributions that are unique to the context of V:N:M sparsity, as elaborated in the main text of our paper. With the proposed three techniques, the novelty of our work also lies in the breadth of scenarios that our framework covers, from visual pretraining to LoRA training and limited-budget training.

2. The original Venom paper has compared sparse matrix multiplication (MM) using the V:N:M-sparse GPU kernel with dense MM performed using CuBLAS. Their results demonstrate that V:N:M-sparse MM significantly outperforms dense MM at high levels of sparsity. For your reading convenience, we list part of the results below. Please note that in the table, "Model" refers to that the shapes of MM are from that model.

Thank you very much for your thoughtful review and constructive feedback. We appreciate your recognition of the strengths of our paper and would like to address the weaknesses you raised.

Q1: Why is the paper claimed that only a 2:4 sparse pattern can be accelerated by sparse tensor cores? I have some doubts about it. One could write a custom CUDA kernel and accelerate based on the indices of the weights.

A1: Thanks for your comment. We would like to clarify our claim by discussing three specific scenarios in which a custom CUDA kernel can accelerate sparse computations based on the weight indices:

1. Unstructured sparsity. When the indices of non-zero weights are completely irregular, a CUDA kernel can only utilize CUDA core instructions to perform sparse matrix-matrix multiplications, which allows for the flexible omission of zero operations.
   • However, it is important to note that the computational power of CUDA cores is significantly lower than that of tensor cores on GPUs. For example, the peak computational power of CUDA cores on an RTX 4090 GPU is 82.6 TFLOPS, while the peak power of tensor cores (with FP16 accumulation) is 330.3 TFLOPS. Given that contemporary dense Transformer models predominantly rely on tensor cores for inference [1], unstructurally sparse Transformers using CUDA cores barely achieve superior inference speeds compared to their tensor core counterparts.
   • Additionally, even when both dense and sparse Transformers execute inference solely on CUDA cores, the irregular computation and memory access patterns associated with unstructured sparsity hinder their ability to maintain competitive inference speeds unless the sparsity exceeds 95% [2]. However, at such high sparsity levels, the accuracy of most Transformers tends to collapse.
2. Block sparsity. In cases where the non-zero weights are organized into blocks—some of which are dense while others are entirely sparse—block-sparse GEMMs can be effectively accelerated using dense tensor cores [2] rather than sparse tensor cores.
3. 2:4 and V:N:M sparsity. When the arrangement of non-zero weight indices adheres to the 2:4 sparse format, i.e., at most two non-zero values among every four consecutive weights, it becomes feasible to utilize sparse tensor cores through the "mma.sp()" instruction within a CUDA kernel to accelerate sparse Transformers. Theoretically, sparse tensor cores offer double the computational capacity of dense tensor cores. Similarly, to comply with the 2:4 sparse format, V:N:M sparsity also implements 2:4 pruning for a weight tensor after vector-wise pruning.

[1] Tensor core usage in CuBLAS.

[2] Accelerating Matrix Multiplication with Block Sparse Format and NVIDIA Tensor Cores

Q3: In Fig.4(a), the authors compare accuracy improvement under similar speedup conditions. In my view, more parameters generally lead to higher accuracy, so it would be fairer to compare accuracy at the same sparsity.

A3: Thank you for your concerns. In our paper, we have also presented substantial results comparing the accuracy at the same sparsity levels. We would like to clarify this further as follows:

1. Table 4 indicates that, using training setting (TS) 1 within our proposed framework, the 64:2:5-sparse DeiT-base achieves a significant accuracy improvement through our V:N:M-specific channel permutation, while maintaining the same parameter count.
2. Table 5 shows that under TS2 in our framework, the 64:2:8-sparse DeiT-base has fewer parameters than related works (22.7M vs. 25.4M) while achieving nearly lossless accuracy compared to its dense counterparts.
3. Table 6 further illustrates that with TS2 in our framework, the 64:2:5-sparse Swin-base attains the same classification accuracy as state-of-the-art models but requires significantly fewer parameters (35.9M vs. 64.1M). Additionally, for object detection, the 32:2:6-sparse H-DETR using TS2 achieves a higher mean average precision (mAP) than fixed mask training (45.3% vs. 44.1%) while utilizing the same number of parameters.
4. Table 7 demonstrates that with TS3 in our framework, the 64:2:5-sparse Llama2-7B, despite having fewer parameters than related works employing 2:4 sparsity, achieves better perplexity on downstream tasks.

Altogether, on one hand, we list tables or figures to demonstrate that our proposed framework can lead to lower parameters or FLOPs over related works for a wide range of V:N:M-sparse Transformers while maintaining comparable or higher accuracy, such as Table 5,6 and 7. On the other hand, we would also like to use tables or figures to illustrate the effectiveness of those individual novel techniques embedded within the framework, such as Figure 4(a), which highlights our V and M selection technique.

Besides, we would like to clarify that for V:N:M-sparse Transformers, "more parameters generally lead to higher accuracy" is not necessarily tenable. To illustrate this, we present a specific example in Table 5.1. Among the three (V, M) configurations (16,16), (32,16), and (128,15) for DeiT-base models, the 128:2:15-sparse DeiT-base has the highest parameter count, yet the 16:2:16-sparse DeiT-base, which exhibits the highest MD, achieves the best accuracy, significantly surpassing the other two configurations. The principle behind the example is that, both the sparse granularity determined by V and the retained parameter counts dictated by M influence the accuracy of V:N:M-sparse Transformers. Compared to relying solely on parameter counts, MD accounts for both factors, thereby providing a more proper measure of the accuracy of V:N:M-sparse Transformers.

Table 5.1. V:N:M-sparse DeiT-Base 30 epoch LoRA accuracy on downstream tasks

Q4: Fig.4(a) shows how V and M selection impacts the accuracy and speedup of V:N:M sparse Transformers. However, the V:N:M setting in Fig.8 differs. Does this imply that V and M selection is not applied in Fig.8?

A4: Yes. Figure 8 in Appendix D of our paper is to detail the speedups achieved by various ViT models under different V:N:M sparsity. It is noteworthy that Figure 8 is not intended to demonstrate the effectiveness of our V and M selection technique. From this question, we sincerely appreciate your careful review of our paper and value your attention to detail.

Thanks for your response, which addressed my concern. After reading my and others' responses, I will increase my rate.

Q1: [Novelty] Differences between this paper and the Venom paper [1]. Wasn't V:N:M proposed in the Venom paper ? How does it differ from Venom ?

A1: Thanks. we would like to clarify that the concept of "V:N:M sparsity" was first introduced in the Venom paper; however, that initial work concentrated on the GPU kernel design for V:N:M-sparse GEMMs. In contrast, our research focuses on improving the accuracy of a wide range of V:N:M-sparse Transformers under different scenarios. Besides, we would like to detail the significantly distinguishable contributions and advancements our work makes beyond the initial V:N:M sparsity study and other related works:

1. Our work identifies and addresses critical issues that have long affected the field of Transformer 2:4 sparsity, yet have received insufficient attention until now. That is, 2:4-sparse Transformers in practice provide very limited end-to-end speedups, while the promising V:N:M sparsity, although capable of delivering higher speedups, has not demonstrated comparable accuracy for sparse Transformers relative to 2:4 sparsity. Our work exactly aims to bridge this gap by proposing a framework with three innovative techniques that algin the accuracy V:N:M sparsity with that of 2:4 sparsity. In doing so, our work significantly contributes to the field by promoting the widespread adoption of V:N:M sparsity as a truly effective solution for compressing Transformers.
2. Our work tackles three key fundamental challenges that prevented V:N:M-sparse Transformer from achieving the desired accuracy—challenges that have never been addressed in previous research. The three challenges are:(1) selecting appropriate values for V and M, (2) enhancing the accuracy of V:N:M sparsity under limited training budgets, and (3) implementing V:N:M-sparse LoRA training for large language models (LLMs). These challenges are critical and cannot be overlooked. Accordingly, our proposed three techniques to address challenges are also novel contributions that are unique to the context of V:N:M sparsity, as elaborated in the main text of our paper. With the proposed three techniques, the novelty of our work also lies in the breadth of scenarios that our framework covers, from visual pretraining to LoRA training and limited-budget training.
3. We further clarify the originality in our paper. we have highlighted key distinctions between our work and the initial V:N:M work in the paper. First, as shown in Figure 2, our work and the initial work for V:N:M sparsity lies in distinct phases in the workflow of compressing a Transformer. The initial work primarily proposes GPU acceleration kernels, i.e., Figure 2(b). In contrast, our research emphasizes enhancing the accuracy of V:N:M-sparse Transformers under various scenarios. Our proposed framework within the three novel techniques is depicted in Figure 2(a). Importantly, combining the three novel techniques with other factors, such as pruning criteria, to form our comprehensive framework is non-trivial. For instance, under the limited training budget, employing RIA-based pruning rather than ABS-based pruning is essential for achieving better accuracy, as demonstrated below.

Table 4.1 Results of 64:2:5-sparse DeiT-base on downstream tasks using RIA and ABS-based pruning, respectively.

Q2: The Venom paper proposed a GPU kernel for speedup sparse models on GPUs. how does this paper handle processing on the GPU? Does it propose a dedicated GPU kernel? If so, what are the differences compared to the Venom [1] GPU kernel?

A2: Thank you. We utilize the same GPU kernel as employed in the Venom paper to accelerate V:N:M-sparse Transformers. Our major contribution is the enhancement of accuracy and applicability of V:N:M-sparse Transformers, as stated before.

Q3: The title of the paper includes "Transformer"—what specific characteristics of Transformers does it take into account?

A3: Thanks. Given that the original V:N:M GPU kernel in the Venom paper only provides speedup for linear layers, we selected Transformers which include a number of linear layers in their structure for employing V:N:M sparsity to obtain high end-to-end speedups. However, your feedback is greatly appreciated and we put the development of V:N:M-sparse GPU kernels for convolutional operations as our future work.

Q4: Looking at Table 5, the caption lacks sufficient explanation. For instance, a clear description of " $\Delta$ " is necessary in the caption.

A4: Thank you very much for your carefulness.

1. In our revised paper, we have linked the caption with the description of $\Delta$ and $\downarrow \%$. For instance, in the case of the DeiT-base model, as shown below, the $\Delta$ means the difference in accuracy between the related works and the dense DeiT-base. The first $\downarrow \%$ indicates the proportion of parameter reduction relative to the parameters of the dense DeiT-base, while the second $\downarrow \%$ signifies the proportion of FLOP reduction compared to the FLOPs of the dense DeiT-base.

2. Furthermore, in Appendix J of our revised paper, we outline the references corresponding to each related work in Table 5 to enhance clarity. For your reading convenience, we also present the list here:

Table 4.2 Details of related works in Table 5 of the main paper

3. Besides, we conducted a thorough review of all tables and figures to ensure clarity and completeness throughout the paper. We sincerely appreciate your valuable feedback.

Thanks the authors for providing the feedback. By checking the rebuttal and comments, i think there are still 'lack of novelty' of this paper. So i maintain my score.

Thank you very much for your feedback. We humbly and respectfully request your renewed attention to our technical novelty and contributions:

1. Our work primarily focuses on algorithmic advancements rather than GPU kernel acceleration. The novelty in our algorithms is noteworthy in both scope and effectiveness. We propose three novel methods for V:N:M-sparse Transformers: V and M selection, V:N:M-specific channel permutation, and three-staged LoRA training method. In particular:

   • For our proposed V and M selection method, we introduce the concept of mask diversity (MD) for V:N:M Transformers for the first time and provide theoretical proof demonstrating that the MD of different V and M configurations can be efficiently compared using relative MD. This relative MD allows for the rapid selection of appropriate V and M values, enabling the corresponding V:N:M-sparse Transformer to achieve optimal accuracy and speedup tradeoffs. The effectiveness of our V and M selection method is thoroughly validated across all training settings, as shown in Figures 4 and 10 of our paper, and across multiple Transformer architectures, as illustrated in Figures 11 (ViT-large), 12 (ViT-huge), and 13 (Llama2-13B) of our paper.

   • For our V:N:M-specific channel permutation method, we provide insights into how both input and output channel permutations of weight matrices can influence the accuracy of V:N:M-sparse Transformers—an aspect that differs significantly from 2:4 sparsity. We model the V:N:M-specific channel permutation as an alternative optimization problem to efficiently solve for the permutation matrices. The effectiveness of our proposed channel permutation method is extensively validated, as demonstrated in Tables 12, 13, 14, and 15 in our paper.

   • In our three-staged LoRA training method for V:N:M-sparse LLMs, we propose, for the first time, that a brief duration of dynamic mask changes following dense LoRA initialization can significantly enhance the effectiveness of LoRA training. This approach optimizes the trade-offs between exploration and exploitation in mask selection. The effectiveness of our three-staged LoRA training method is validated through Table 8 and Figure 5 in our paper.

2. Our innovations help address three critical challenges that have never been investigated and addressed by previous research: the selection of appropriate V and M values for V:N:M-sparse Transformers, the enhancement of accuracy for V:N:M-sparse Transformers under limited training budgets, and the integration of V:N:M sparsity into LoRA to facilitate the fine-tuning of V:N:M-sparse LLMs. More importantly, through these innovations, our framework achieves that the accuracy of a wide range of V:N:M-sparse Transformers is comparable to or exceeds that of 2:4-sparse Transformers. Given that V:N:M-sparse Transformers also offer enhanced speedup compared to their dense counterparts and exhibit superior speedup-accuracy scalability, our work represents a significant advancement in establishing V:N:M sparsity as atruly viable alternative to 2:4 sparsity for compressing highly redundant Transformers.

Q5: In section 4.3, the definitions of $Mv$, $Wv$, $Bv$, and $Av$ are unclear. Also, it would better to disclose detailed LoRA configurations in the three-staged LoRA training experiments (section 5.4) , such as dynamic sparse mask initialization, the interval of updating sparse masks, the actual training iteration assignment two different stages etc.

A5: Thanks. The $Mv$, $Wv$, $Bv$, and $Av$ are typos and should be corrected as $M$, $W$, $B$, and $A$, respectively. We have made these revisions throughout the paper to ensure consistency and clarity. We apologize for the oversight and sincerely appreciate your carefulness.

Besides, we are very pleased to detail the configurations used in our three-stage LoRA training as outlined in Table 2.3:

Table 2.3 the details of our three-staged LoRA training

Specifically,

• To obtain the initial dynamic sparse masks $M$ for the weight matrix $W$ in the second stage of LoRA training, we first merge the LoRA matrices $BA$ with $W$. RIA-based pruning is then applied to the merged matrix, where the retained weights are accordingly assigned a value of 1 in $M$, while the pruned weights are assigned a value of 0 in $M$.
• We set the interval for updating sparse masks to 20 iterations. A smaller interval can destabilize LoRA training, a phenomenon also noted in DeiT-base in our paper (Please refer to Table 1 in our paper).
• We adjust hyperparameters, including the mask update intervals and update counts, to ensure that the first, second, and third stages account for approximately 2.5%, 2.5%, and 95% of the total training, respectively.
• Following standard practice, we use 1,024 tokens for each training iteration, resulting in a total of 12 million tokens for our LoRA training.
• The ranks of the LoRA matrices, specifically $A$ and $B$, are set to 16, with LoRA $\alpha$ configured to 32 to maintain the regular setting of $\alpha/\text{rank} = 2$.
• All the linear layers in Llama2 are equipped with LoRA training.

Q8: Can channel permutation enhance the performance of SR-STE training or LoRA training?

A8: Yes.

1. Our V:N:M-specific channel permutation (CP) does enhance the LoRA training performance. For instance, as shown in Table 3 in the main text of our paper, our CP improves the 5-shot score of Llama2-7B under low LoRA training budgets by 1.25%. In visual tasks, our CP increases the accuracy of the 64:2:5-sparse DeiT-base during LoRA training on downstream tasks by 1.63% (Table 4 in the main text of our paper). Moreover, the improvements resulting from our CP are consistent across different V:N:M ratios, as demonstrated in Tables 2.6, 2.7, and 2.8 below.
2. The influence of CP on SR-STE training is not pronounced in our experiments.

Table 2.6 Results of 64:2:16 DeiT-base on downstream tasks with 30 training epochs

Table 2.7 Results of 32:2:16 DeiT-base on downstream tasks with 30 training epochs

Table 2.8 Results of 128:2:15 DeiT-base on downstream tasks with 30 training epochs

Q9: How is the regularization coefficient $\lambda$ in equation (3) tuned to improve training stability?

A9: Thank you very much. We utilize the method described in [4] to tune $\lambda$. Specifically, we first perform a grid search to compute the flip rate for each candidate $\lambda$. Subsequently, we select a $\lambda$ with a "healthy" flip rate as defined in [4] for our training process.

[4]: Accelerating Transformer Pre-training with 2:4 Sparsity. ICML2024.

Q7: For large language models (LLMs), latency is not solely determined by FLOPs; it is typically constrained by memory bandwidth and access patterns. In Definition 1, is the measured speedup based on a batch size of 1? Furthermore, when varying the batch size, will this lead to different V and M sparsity values?

A7: Thanks.

1. The speedup results presented in our paper are based on a batch size of 1.
2. When varying the batch size, the selected values for V and M differ accordingly. For example, the results for Llama2-7B at various batch sizes are shown in Table 2.5, where the selected V and M values are highlighted in bold for clarity. The speedup thresholds are randomly generated from the range [1, 2.5]. Since the speedup of a V:N:M-sparse Transformer compared to its dense counterpart varies with different batch sizes, the selection results will also differ. We consider this phenomenon to be normal and it does not impact our technical contributions. In practical LLM inference systems, especially in cloud environments, multiple queries are often concatenated into a fixed batch before inference, which scenario is still suitable for the application of our techniques.

Table 2.5 Selected V and M values of Llama2-7B under different batch sizes (BS). (X,4) means 2:4 sparsity

Q6: Figure 3 does not intuitively illustrate "in the absence of regularization, frequent updates to the masks can negatively impact the gradient flow of V:N:M sparse Transformers during fine-tuning". Additionally, adding a comprehensive empirical study would make it more convincing to draw the conclusion that "the iterations for the first two stages should not exceed 10% of the total iterations."

A6: Thanks.

1. We would like to clarify that gradient flow, in our context, specifically refers to the change in gradient norms during the LoRA training process. As shown in Figure 3 in the main paper, the gradient norms for LoRA with dynamic masks are consistently lower than those with fixed masks. Such low gradient norms may lead to underfitting in neural networks, undermining the gains achieved through mask exploration and negatively impacting accuracy [3].
2. To further intuitively illustrate the necessity for infrequent mask updates, we present the loss change curves for sparse LoRA fine-tuning with both fixed and dynamic masks. As depicted in Figure 2.4 below, constant mask updates result in a progressively higher training loss compared to the fixed-mask training as the training progresses.

Figure 2.4 Loss changes of Llama2-7B during sparse LoRA fine-tuning with dynamic masks and fixed masks, respectively.

[3] Training your sparse neural network better with any mask. ICML2022.

3. We conducted experiments with Llama2-7B to assess the impact of the proportion of the first two stages relative to total iterations. As shown in Table 2.4, a proportion of 5% yields the best results for both 64:2:5 and 64:2:6 configurations. Moreover, when the proportions exceed 10%, the results become unstable.

Table 2.4 5-shot average scores for the different proportions of the first two stages in Llama2-7B LoRA training

Q2: In Definition 1, the objective is to maximize accuracy under speedup constraints s. However, there should be experimental evidence demonstrating that the chosen V and M values meet these constraints.

A2: Thank you.

1. Using Llama2-7B as an example, we have listed the selected V and M values for different batch sizes, along with the corresponding speedups of the V:N:M-sparse Llama2-7B compared to their dense counterpart. This information is presented in Table 2.5 (located in the context of part (Q7)). We randomly generated multiple speedup constraints s from the interval [1, 2.5], resulting in s∈{1.14, 1.26, 1.34, 1.52, 1.65, 1.88, 2.12}. From the table, it is evident that our proposed V and M selection method yields (V, M) pairs that consistently achieve speedups exceeding the corresponding constraints. For instance, with a batch size of 8 and a speedup constraint of 1.52, the selected 128:2:6-sparse Llama2-7B achieves a speedup of 1.55, which is greater than 1.52.
2. Additionally, in figures such as Figure 4 and Figure 10 in our paper, the x-axis of the blue line dots represents the speedup constraints, while the x-axis of the yellow line dots represents the speedups achieved by the selected (V, M) pairs. It is clear that the yellow lines consistently lie above and to the right of the corresponding blue lines, indicating that our chosen V and M values satisfy the speedup constraints.
3. Finally, we further provide the V and M selection results for ViT-large, ViT-huge, and Llama2-13B for your reference, as shown in Figures 2.1, 2.2, and 2.3, respectively. Our V and M selection method performs effectively, and the selected values meet the speedup constraints across all three models as well.

Figure 2.1 Average Top-1 accuracy of ViT-large on downstream tasks with different V and M vlaues

Figure 2.2 Average Top-1 accuracy of ViT-huge on downstream tasks with different V and M vlaues

Figure 2.3 Average scores of Llama2-13B on 5-shot tasks with different V and M values.

Q3: In Section 5.4, although V:N:M sparse Llama2 outperforms RIA and Wanda, it still suffers significant accuracy loss compared to the dense counterpart. For instance, HellaSwag accuracy drops from 57.23 to 42.88, and ARC-C from 43.3 to 34.3.

A3: Thank you for your feedback. The accuracy drop compared to the dense counterpart is primarily due to limited computational resources. In our study regarding Llama2, we conduct experiments with at most 12 million tokens. This amount is far from sufficient to restore accuracy to the levels achieved by the dense model. Despite this limitation, our results still notably surpass those of related works on post-training 2:4 sparsity in the field.

However, we would like to highlight that given adequate training, sparse large language models (LLMs) can achieve performance levels comparable with dense counterparts, while still benefiting from the substantial speed advantages associated with sparsity. For instance, [1] utilized 7.5 billion tokens for continual training of the 2:4-sparse Llama2-7B, achieving scores of 55.24 on HellaSwag and 41.11 on ARC-C. Furthermore, [2] reported that with 145 billion tokens and sparse pretraining, which setup is typically prohibitively costly for most researchers, the 70%-sparse Llama2-7B can fully recover accuracy for fine-tuning tasks.

[1]. Pruning large language models with semi-structural adaptive sparse training. ARXIV2024.

[2]. Enabling high-sparsity foundamental Llama models with efficient pretraining and deployment. ARXIV2024.

Q4: Equation (3) lacks the description of $\overline{M}_{t-1}$.

A4: Many thanks. In the newly uploaded version of our paper, we have included the explanation of $\overline{M}{t-1}$, clarifying that it represents the logical negation of the mask $M{t-1}$ at training step $t-1$. This mask identifies the weights that have been pruned, allowing the regularization term to focus exclusively on these weights, thereby facilitating a gradual reduction in their norms. Consequently, after sufficient training, the norms of the pruned weights in a weight matrix approach zeros, resulting in the matrix conforming to the desired sparse pattern. Additionally, we have thoroughly revised the paper to ensure that all notations in the formulas are accompanied by appropriate descriptions.

We are very grateful for your recognition and constructive feedback. We aim to address the identified weaknesses and questions systematically.

Q1: While it's intuitive that a smaller M results in lower sparsity in the sparse transformers, the process for excluding certain (V, M) combinations is unclear. Additionally, there is insufficient evidence in the paper demonstrating that mask diversity (MD) improves transformer accuracy.

A1: Thank you very much for your comments.

1. To clarify the V and M selection process, we provide an example in Table 2.1. Assuming a speedup threshold of 2, the two-phase sifting is as follows:
   • Coarse-Grained Sifting: For each row of V (16, 32, 64, 128) in Table 2.1, we first select the minimal M that achieves a speedup greater than 2. For instance, for V=16, M=16 is chosen, while M values of 17, 18, and 19 are excluded since their corresponding accuracy for V:N:M-sparse Transformers is lower than that with M=16, which retains more parameters. Similarly, for V=32, 64, and 128, we select M values of 16, 15, and 15, respectively. The speedups for these four (V, M) pairs—(16, 16), (32, 16), (64, 15), and (128, 15)—are highlighted in bold in Table 2.1 and exceed the threshold of 2. After coarse-grained sifting, most of the (V, M) pairs that also meet the speedup threshold are excluded, retaining only those that are critical with respect to the threshold.
   • Fine-Grained Sifting by Mask Diversity (MD): We compute and compare the MDs of the four selected (V, M) pairs. The pair with the highest MD is selected for constructing the V:N:M-sparse Transformers.

Table 2.1 A example illustrating the V and M selection process

2. To further demonstrate that MD, rather than model parameter counts, serves as a better indicator for V:N:M-sparse Transformers, we present a specific example in Table 2.2. Among three (V, M) configurations, the 128:2:15-sparse DeiT-base has the highest parameter count, yet the 16:2:16-sparse DeiT-base, which exhibits the highest MD, achieves the best accuracy, significantly surpassing the other two configurations. The principle behind the example is that, both the sparse granularity determined by V and the retained parameter counts dictated by M influence the accuracy of V:N:M-sparse Transformers. Compared to relying solely on parameter counts, MD accounts for both factors, thereby providing a more accurate measure of performance.

Table 2.2 V:N:M-sparse DeiT-Base 30 epoch LoRA accuracy on downstream tasks

Q3: The work would be stronger if it provided performance benchmarks for Llama2-7B on pretraining benchmarks, such as MMLU or GSM-8k, to give a more comprehensive evaluation of the model's capabilities across a broader set of important benchmarks. If computational resources are limited, consider performing post-training on Llama2-7B to demonstrate the applicability of this paper's methods.

A3: Thank you very much for your suggestion. As you noted, GSM8K and MMLU serve primarily as pretraining benchmarks, actually requiring extensive pretraining or continuous training of sparse LLMs with billions of tokens prior to evaluation. In contrast, our three-staged LoRA training method is specifically designed for post-pretraining fine-tuning using millions of tokens. Nevertheless, we sincerely appreciate your suggestion and evaluated the performance of 64:2:5-sparse Llama-7B on the GSM-8K benchmark by post-pretraining fine-tuning using our proposed method. The results are presented in Table 1.2 below.

The results indicate that the mathematical capabilities of Llama2-7B are significantly compromised by sparsity, both in the 2:4 and V:N:M configurations. However, our 64:2:5-sparse Llama2-7B still outperforms the post-trained 2:4-sparse counterpart. Notably, even after continual training with 7.5 billion tokens, the 2:4-sparse Llama2-7B still exhibits a score that is over 4 points lower than that of the dense model on GSM-8K [2]. It is important to highlight that our LoRA training for the sparse Llama2-7B utilized only 12 million tokens.

Regarding the MMLU benchmark, both the 2:4-sparse and 64:2:5-sparse Llama2-7B scores are close to random guessing, at approximately 25%, following LoRA training only with 12 million tokens. Generally, for knowledge-intensive tasks, N:M sparsity, including V:N:M sparsity, demonstrates ineffectiveness with limited tokens [3]. We consider this an area for future research.

[2]. Pruning large language models with semi-structural adaptive sparse training. ARXIV2024.

[3]. Compressing LLMs: the truth is rarely pure and never simple. ICLR2024.

Q1: The paper lacks a comparison with the latest related works, which limits its ability to contextualize the advantages and innovations within the current research landscape, such as [1]. Moreover, please also report the model performance without sparsity in Figure 4,5,7,8,9 to show the influence of introducing model sparsity.

A1: Thank you very much for your constructive feedback.

1. We have included [1], referred to as NViT, in Table 5 of our newly revised paper as one of the related works for comparison. Here, we provide further details on the comparison: 1) In terms of accuracy, both our method and NViT achieve nearly lossless Top-1 accuracy for sparse DeiT-base and DeiT-small. 2) Regarding FLOPs reduction, our method achieves the highest reduction for both DeiT-base (71.6% ↓) and DeiT-small (54.3% ↓) when compared with NViT and other related works. 3) For parameter reduction, our method achieves the highest reduction for DeiT-small (57.9% ↓) among related works, while NViT achieves the highest reduction for DeiT-base (80.4% ↓), compared to our 73.8% ↓. It is noteworthy that NViT combines multiple strategies for compressing DeiTs, including global structural pruning, 2:4 pruning, and parameter redistribution. In contrast, our work focuses solely on V:N:M sparsity and has the potential to further enhance parameter reduction ratios when combined with other compression strategies. 4) For speedups, both NViT and our method yield significant practical speedups for sparse DeiTs. Specifically, NViT-B with ASP achieves speedups of 1.86x and 1.85x on V100 and RTX 3080 GPUs, respectively, while our 64:2:8-sparse DeiT-base achieves a 2.08x speedup on RTX 3090 GPUs.
2. We have added the performance results without sparsity in Figures 4, 5, 7, 8, and 9 in our revised paper. Particularly for Llama2-7B, we present results for both the dense model and the 2:4-sparse model; the latter serves as the primary benchmark for comparing the performance of our V:N:M-sparse Llama2. Please note that Figure 7 has been re-numbered as Figure 10 in the latest version of our paper.

Q2: The experiments in this paper are limited to small-scale models, such as ViT, DeiT, and Llama-7B. It does not extensively explore the performance on larger models, like ViT-Huge or LLaMA models with higher parameter counts, where the effectiveness of the sparsity method might vary.

A2: Thank you for your concern. We endeavor to provide four results on larger models:

1. For LLM LoRA training, we present the average scores of Llama2-13B on 5-shot tasks with different V and M values, as depicted in Figure 1.1. The average score for the dense model is 68.23, while the average score for the 2:4-sparse model is 56.78.
2. For visual downstream tasks, we report the average Top-1 accuracy for ViT-large and ViT-huge with different V and M values. These are illustrated in Figure 1.2 and Figure 1.3, respectively. The average Top-1 accuracy for the dense ViT-large is 94.53%, while for the dense ViT-huge, it is 92.99%.

Figures 1.1, 1.2, and 1.3 demonstrate that Training Setting 1 (TS1) and Training Setting 3 (TS3) in our proposed framework, along with the novel techniques integrated within it, perform effectively for larger models.

3. Regarding approximate visual pretraining, we benchmark the accuracy of the 64:2:8-sparse ViT-large trained on ImageNet1K, as shown in Table 1.1 below. The 64:2:8-sparse ViT-large does not fully recover the accuracy of its dense counterpart due to non-strict pretraining pipelines. Specifically, the dense ViT-large is pretrained on ImageNet21k using TPUv3 hardware and subsequently fine-tuned on ImageNet1K, achieving a Top-1 accuracy of 84.31%. However, pretraining the 64:2:8-sparse ViT-large on ImageNet21k is challenging with our limited computational resources and infrastructure. Nevertheless, our 64:2:8-sparse ViT-large achieves a Top-1 accuracy of 81.24% when only training on ImageNet1K with a speedup of 1.88x compared to the dense model.

Table 1.1 Accuracy of 64:2:8-sparse ViT-large on ImageNet1K

Thanks for your response.

Based on the post-training results for the Llama models, it seems that V:N:M sparsity is not well-suited for Llama model training. While it does show improvements compared to the baseline, the accuracy loss can not be neglected relative to the 1.5x speedup it provides.

I will maintain my current score, but I encourage you to continue optimizing your work further. Best of luck!

Thanks for your response.

1. Why does NViT-B(ASP) achieve better Top-1 accuracy than the original dense model in Table 5?

• For instance, NViT-B(ASP) reports a Top-1 accuracy of 83.29 for DeiT-B (compared to the dense model's 81.84) and 82.19 for DeiT-S (compared to the dense model's 79.85).
• Could you provide further explanations for this result? Were the baseline results for the dense models taken directly from other papers or re-evaluated in your experiments?

2. The reported accuracy for Llama-7B-dense is 61.99 (Figure 4), while Llama-13B-dense achieves 68.23 (Figure 8). Given that Llama-7B-dense has approximately 2x the inference efficiency of Llama-13B-dense, the V:N:M sparse models in Figure 8 exhibit lower accuracy than Llama-7B-dense, with smaller speedup gains compared to 7B. Does this imply that the V:N:M sparsity approach is less/not effective for Llama models?

3. Based on current results, the V:N:M sparsity approach demonstrates better suitability for DeiT and ViT models, where the performance trade-offs are more favorable.

Thank you very much for your concern. We appreciate the opportunity to provide further explanation.

Explanation 1: The improved Top-1 accuracy of NViT-B(ASP) and NViT-S(ASP) compared to the original dense model, is due to knowledge distillation (KD), which is explicitly acknowledged in the NViT paper.

• For a fair comparison, NViT-B(ASP) and NViT-S(ASP) should be compared to the dense model also under the same KD conditions to derive the accuracy difference (represented as $\Delta$ in our paper). For instance, the accuracy of the dense DeiT-base models under KD is 83.6%. In Table 5, We exactly use the accuracy of the KD-based dense models to calculate $\Delta$. Besides, in our paper, $\Delta$ for NViT is marked with an asterisk to indicate its specific computation method, detailed in Appendix J.
• It is important to note that KD is generally considered parallel to other model compression methodologies, such as sparsification (including V:N:M sparsity) and quantization. Considering this, we use the results of the pure dense model without KD as our baseline, allowing for a clearer evaluation of the individual efficacy of our V:N:M sparsification method. This manner of using a pure dense model without KD as a baseline is more widely adopted in the literature.
• The baseline results without KD are identical to those reported in the original DeiT paper [4] and have been accurately reproduced in our experiments.

[4] Training data-efficient image transformers & distillation through attention. ICML2021.

Explanation 2: The absolute accuracy of sparse Llama models, including Llama2-7B and Llama2-13B, can be significantly enhanced on benchmarks such as question-answering tasks and GSM8K through increased training token counts coupled with adequate training, such as pretraining and continual training.

• In addition to the previously mentioned study [2], which employs 7.5 billion tokens for continual training of sparse Llama2-7B to narrow the accuracy gap to 4.1% compared to its dense counterpart on GSM8K tasks, another study [5] utilizes 145 billion tokens to pretrain a sparse Llama2 backbone, achieving full recovery of fine-tuning accuracy for tasks including GSM8K. These findings suggest the considerable potential of sparse LLMs. Meanwhile, this level of resource requirements is typically prohibitive for general research.

• We would like to clarify that our three-staged LoRA training method is specifically designed for post-pretraining fine-tuning using millions of tokens, which for the first time enables V:N:M-sparse LLMs to outperform their post-training 2:4 counterparts in a highly data-efficient manner. We kindly deem that optimizing absolute accuracy on pretraining or continual training-dependent benchmarks like GSM8K and MMLU is beyond the scope of our current work and is better suited for future research.

Thank you so much for your empathetic understanding and valuable reviews!

[5]. Enabling high-sparsity foundamental Llama models with efficient pretraining and deployment. ARXIV2024.

Q3："V:N:M-specific channel permutation, which improves the accuracy of V:N:M-sparse Transformers". Why channel permutation can improve accuracy?

A3: Thank you for your insightful question regarding the function of channel permutation.

1. Channel permutation (CP) is based on a property of neural networks: parameter space symmetries [1]. That is, different parameter configurations can yield identical loss values for a neural network. This loss-invariant transformation to a neural network is also called teleportation [2]. CP is exactly a special form of teleportation.
2. CP can change the training dynamics [3]. In our work, pruning after conducting V:N:M-specific CP can lead to less loss in weights, which means more important weights, like outlier weights [4] are probably retained. Hence, with our V:N:M-specific CP, the V:N:M-sparse Transformers upon pruning usually possess higher accuracy, which also benefits the subsequential training. Notably, CP is especially essential in scenarios given limited training budgets.
3. To further illustrate the effectiveness of our V:N:M-specific CP, we present additional results regarding CP performance under various V:N:M ratios in the limited-training-budget scenario, as shown in Tables 3.2, 3.3, and 3.4 below. The results clearly indicate that our CP method significantly enhances the accuracy of these V:N:M-sparse Transformers.

Table 3.2 Results of 64:2:16 DeiT-base on downstream tasks with 30 training epochs

Table 3.3 Results of 32:2:16 DeiT-base on downstream tasks with 30 training epochs

Table 3.4 Results of 128:2:15 DeiT-base on downstream tasks with 30 training epochs

[1] Improving convergence and generalization using parameter symmetries. ICLR 2024

[2] Neural Teleportation. Mathematics 2023, 11(2), 480

[3] Level Set Teleportation: An Optimization Perspective. Arxiv 2024.

[4] Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. ICML 2024.

Q4："a higher MD leads to better Transformer accuracy" Is this claim from extensive experiments or other places?

A4: Thank you for your question regarding the relation between MD and Transformer accuracy.

1. The original opinion that the structure constraints of sparse patterns cause accuracy degradation is from the paper [5], where the authors employ mask diversity (MD) to quantify the structure constraints of a sparse pattern. They have validated in ResNet models that a higher MD leads to better accuracy.
2. However, [5] does not give the MD of the V:N:M sparse pattern, nor does it employ MD in Transformers, which is non-trivial. Instead, we propose Formula 5 in the paper to determine MD for V:N:M-sparse Transformers. Furthermore, we prove that for the same Transformer, the relative order of MD of different (V,M) configurations is irrespective of the Transformer weight shapes. This finding largely simplifies the computation of MD of V:N:M-sparse Transformer, which facilitates the efficient execution of the proposed V and M selection techniques.
3. Our experiments empirically show that a higher MD leads to better accuracy also holds for V:N:M-sparse Transformers. In particular, besides the results in the main paper, we also present a concrete example as Table 3.5 below. Among three (V,M) configurations, although 128:2:15-sparse DeiT-base has the most parameters, the 16:2:16-sparse DeiT-Base which has the highest MD also has the highest accuracy with a significant gap beyond the other two V:N:M-sparse Transformers.

Table 3.5. V:N:M-sparse DeiT-Base 30 epoch LoRA accuracy on downstream tasks

[5] Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. NeurIPS 2021.

Q1：The heuristic method for selecting V and M values may not work for extensive Transformers models.

A1: Thank you for your kind concerns. We have extended our proposed V and M selection method to three additional representative Transformer models: ViT-large, ViT-huge, and Llama2-13B. The experimental results demonstrate that V:N:M-sparse Transformers utilizing our selection method consistently achieve superior accuracy-speedup trade-offs compared to those that do not employ this approach. It is clear that our V and M selection method can better scale the V:N:M-sparse Transformers.

Please refer to the figures below for detailed results:

Figure 3.1 Average Top-1 accuracy of ViT-large on downstream tasks with different V and M vlaues

Figure 3.2 Average Top-1 accuracy of ViT-huge on downstream tasks with different V and M vlaues

Figure 3.3 Average scores of Llama2-13B on 5-shot tasks with different V and M values.

Q2: The novelty and originality may not be enough. Authors may need to explain originality more.

A2: Thanks. We understand your concern about the novelty and originality of our work. We would like to emphasize the significantly distinguishable contributions and advancements our work makes beyond the initial V:N:M sparsity study and other related works.

1. our work identifies and addresses critical issues that have long affected the field of Transformer 2:4 sparsity, yet have received insufficient attention until now. That is, 2:4-sparse Transformers in practice provide very limited end-to-end speedups, while the promising V:N:M sparsity, although capable of delivering higher speedups, has not demonstrated comparable accuracy for sparse Transformers relative to 2:4 sparsity. Our work exactly aims to bridge this gap by proposing a framework with three innovative techniques that align the accuracy V:N:M sparsity with that of 2:4 sparsity. In doing so, our work significantly contributes to the field by promoting the widespread adoption of V:N:M sparsity as a truly effective solution for compressing Transformers.
2. Our work does not simply present an engineering implementation, but rather tackles three key fundamental challenges that prevented V:N:M-sparse Transformer from achieving the desired accuracy—challenges that have never been addressed in previous research. The three challenges are:(1) selecting appropriate values for V and M, (2) enhancing the accuracy of V:N:M sparsity under limited training budgets, and (3) implementing V:N:M-sparse LoRA training for large language models (LLMs). These challenges are critical and cannot be overlooked. Accordingly, our proposed three techniques to address challenges are also novel contributions that are unique to the context of V:N:M sparsity, as elaborated in the main text of our paper. With the proposed three techniques, the novelty of our work also lies in the breadth of scenarios that our framework covers, from visual pretraining to LoRA training and limited-budget training.
3. We further clarify the originality in our paper. we have detailed and highlighted key distinctions between our work and the initial V:N:M work in the paper. First, as shown in Figure 2, our work and the initial work for V:N:M sparsity lies in distinct aspects for compressing a Transformer. The initial work primarily proposes GPU acceleration kernels for V:N:M-sparse general matrix multiply (GEMM) operations, i.e., Figure 2(b). In contrast, our research emphasizes enhancing the accuracy of V:N:M-sparse Transformers under various scenarios. Our proposed framework within the three novel techniques is depicted in Figure 2(a). Importantly, combining the three novel techniques with other factors, such as pruning criteria, to form our comprehensive framework is non-trivial. For instance, under the limited training budget, employing RIA-based pruning rather than ABS-based pruning is essential for achieving better accuracy, as demonstrated below.

Table 3.1 Results of 64:2:5-sparse DeiT-base on downstream tasks using RIA and ABS-based pruning, respectively.