Q-Sparse: All Large Language Models can be Fully Sparsely-Activated

We sincerely thank the reviewer for the insightful comments and questions, we would like to give detailed responses to your comments below.

Q1: Inference speedup of Q-Sparse.

A1: Thanks for your insightful question. We measure the inference speedup of Q-Sparse under different sparsity levels and the dense baseline. We benchmark the end-to-end single-batch decoding latency (token/s) by integrating it with the TEAL's kernel [1] on NVIDIA A6000.

As shown in the above table, Q-Sparse brings up to 1.71x, 1.94x, 2.25x speedup compared with the dense model when the sparsity is 40% (TopK=60%), 50% (TopK=50%) and 60% (TopK=40%), respectively.

Q2: Inference speedup in batched inference.

A2: Thanks for your valuable question. Q-Sparse focuses more on single-batch scenarios. For batched inference, we introduce Block Q-Sparse in Section 2.1, which leverages N:M sparsity, where N out of M consecutive elements to be zero, making it more suitable for large-batch inference with an optimized kernel. As presented in Figure 5(c) and Table 2, Block Q-Sparse achieves comparable performance to the dense baselines and Q-Sparse.

The intrinsic balance property of N:M sparsity inherently ensures optimal and complete workload distribution during matrix partitioning for parallel computation. Furthermore, the N:M sparsity constraint on the sparse activations minimizes the locality of memory accesses to the dense weight, enhancing parallel thread efficiency on GPUs.

We benchmark the latency (ms) of matrix multiplication with cuBLAS which is widely used for dense models, and the nmSPARSE-VW32 [2] on NVIDIA A100 GPU. The weight's shape is 1024 x 1024.

As shown in above tables, nmSPARSE-VW32 achieves a speedup of 1.58x in 50% sparsity ratio and 2.94x in 75% sparsity ratio, when the weight's shape is 1024 x 1024 and the batch size is 4096.

Since LLM decoding for half-precision models is memory-bound, quantization can be applied to reduce the time required to transfer model weights at each step, which is orthogonal to activation sparsity. Additionally, decoding with a 1.58-bit LLM is compute-bound rather than memory-bound, allowing it to be combined with Block Q-Sparse to further enhance inference performance.

Q3: Whether the calculation of FLOPs considers Top-K.

A3: We only consider the FLOPs of matrix multiplication, which is a major bottleneck for language models. For example, Given a linear projection $WX$, where $W$ has a shape of $d \times d$ and $X$ has a shape of $d \times 1$, the FLOPs for the matrix multiplication are $2d^2$, while the FLOPs for the top-k operation are $O(d \log d)$.

[1] Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., & Athiwaratkun, B. (2024). Training-free activation sparsity in large language models. arXiv preprint arXiv:2408.14690.

[2] Lin, Bin, et al. "Efficient gpu kernels for n: M-sparse weights in deep learning." Proceedings of Machine Learning and Systems 5 (2023): 513-525.

We sincerely thank the reviewer for the valuable feedback and questions, we would like to give detailed responses to your feedback below.

Q1: Definition of activation sparsity.

A1: Given a linear projection $WX$, where $W$ is the weight matrix and $X$ is the input, activation sparsity is defined as the proportion of zero entries in $X$. We will add the definition in the revised version of our paper.

Q2: The guiding significance of experimental findings in the paper.

A2: Thank you for your question. Q-Sparse enables fully sparse activation in LLMs, significantly improving inference efficiency. To the best of our knowledge, our work is the first to systematically investigate the quantitative relationship between the performance of LLM, activation sparsity level, and model size. We find that as model size increases, the performance gap between fully sparsely activated and dense LLMs narrows progressively. Furthermore, the relationship between the loss and model size follows a power-law, while the relationship between the loss and sparsity level follows an exponential law.

Additionally, Q-Sparse is effective across pre-training, continued training and fine-tuning for off-the-shelf dense LLMs. As shown in Table 1 and Table 2, Q-Sparse has comparable performance to dense models while being more efficient at inference time. It can also be combined with the quantization (e.g., 1.58-bit LLMs) to further enhance the inference efficiency of LLMs.

Q3: More comparative experiments with different models under different architectures on different datasets.

A3: Thanks for your suggestion. Due to the large cost of continue-training, we compare Q-Sparse to ReLUfication and dReLU Sparsification in supervised fine-tuning. Following the setup shown in Section 4.3 of our paper, we finetune the base model of Mistral 7B and Qwen1.5 7B on Open-Orca dataset with ReLUfication and dReLU Sparsification.

As shown in the above tables, Q-Sparse significantly outperforms ReLUfication and dReLU Sparsification in terms of both sparsity and performance. It indicates that ReLUfication and dReLU Sparsification require large amount of training data to make the model adapt to sparse activations, thus are not suitable for fine-tuning.

We would like to thank you for your time and constructive suggestions, we would like to give detailed responses to your comments below.

Q1: Inference speedup of Q-Sparse.

A1: Thanks for your constructive question. We further conduct experiments to measure the inference speedup of Q-Sparse under different sparsity level and the dense baseline. We benchmark the end-to-end single-batch decoding latency (token/s) by integrating it with TEAL's kernel [1] on NVIDIA A6000.

As shown in the above table, Q-Sparse brings up to 1.71x, 1.94x, 2.25x speedup compared with the dense model when the sparsity is 40% (TopK=60%), 50% (TopK=50%) and 60% (TopK=40%), respectively.

Q2: Add evaluation for open-ended generation (e.g., Math or Code).

A2: Thanks for your question. We finetune the base model of Mistral 7B and Qwen1.5 7B on MetaMath dataset [2]. The batch size is set as 128. The learning rates are selected from {3e-6, 5e-6}. All models are trained with 1 epoch for a fair comparison. We evaluate 5-shot GSM-8K using LM Harness.

As shown in the above tables, Q-Sparse can be used to finetune a dense pretrained model to a sparse model with almost no loss at accuracy on GSM-8K dataset.

[1] Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., & Athiwaratkun, B. (2024). Training-free activation sparsity in large language models. arXiv preprint arXiv:2408.14690.

[2] Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., ... & Liu, W. (2023). Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.

We thank the reviewer for the valuable comments and questions, we would like to give detailed responses to your comments below.

Q1: Comparison between MoE and Q-Sparse.

A1: Our approach is orthogonal to MoE and can be combined with it to further improve inference efficiency. MoE models maintain a large expert pool, where only a small number of experts are activated for each token. Q-Sparse extends this sparsity to every linear projection. During matrix multiplication, Q-Sparse prunes entries with smaller magnitudes for each token, thereby reducing the FLOPs during inference.

Q2: Inference speedup in batched inference.

A2: Thanks for your valuable question. Q-Sparse focuses more on single-batch scenarios. For batched inference, we introduce Block Q-Sparse in Section 2.1, which leverages N:M sparsity, where N out of M consecutive elements to be zero, making it more suitable for large-batch inference with an optimized kernel. As presented in Figure 5(c) and Table 2, Block Q-Sparse achieves comparable performance to the dense baselines and Q-Sparse.

The intrinsic balance property of N:M sparsity inherently ensures optimal and complete workload distribution during matrix partitioning for parallel computation. Furthermore, the N:M sparsity constraint on the sparse activations minimizes the locality of memory accesses to the dense weight, enhancing parallel thread efficiency on GPUs.

We benchmark the latency (ms) of matrix multiplication with cuBLAS which is widely used for dense models, and the nmSPARSE-VW32 [1] on NVIDIA A100 GPU. The weight's shape is 1024 x 1024.

As shown in above tables, nmSPARSE-VW32 achieves a speedup of 1.58x in 50% sparsity ratio and 2.94x in 75% sparsity ratio, when the weight's shape is 1024 x 1024 and the batch size is 4096.

Since LLM decoding for half-precision models is memory-bound, quantization can be applied to reduce the time required to transfer model weights at each step, which is orthogonal to activation sparsity. Additionally, decoding with a 1.58-bit LLM is compute-bound rather than memory-bound, allowing it to be combined with Block Q-Sparse to further enhance inference performance.

Q3: Discussion about why gradients should be maintained for non-activated neurons

A3: We presents the gradients and loss curves of Q-Sparse with and without STE in Figure 2 and Figure 6, respectively. Adding STE significantly alleviates the issue of smaller gradients in the lower layers caused by activation sparsity, thus accelerating the model's convergence.

[1] Lin, Bin, et al. "Efficient gpu kernels for n: M-sparse weights in deep learning." Proceedings of Machine Learning and Systems 5 (2023): 513-525.