R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference

Many thanks to Reviewer sWLw for the constructive suggestions, which have helped us further improve the quality of our work. Below, we provide detailed responses to address the concerns.

[Q1: Connection between the motivation and the method] Thanks for the question. We’d like to clarify that the motivation for Case I (Non-sparse components as biases) primarily serves as an initial investigation for Case II (Rank-aware activation sparsity). The observation of rank-aware activation sparsity then directly supports the R-Sparse method.

As discussed in Line 200, obtaining the non-sparse bias terms is computationally expensive, while each individual bias term only contributes a rank-1 component. This motivated us to investigate the overall rank across multiple bias terms from different inputs. We discovered that the space spanned by these biases across thousands of tokens exhibits a low-rank structure, highlighting the relationship between low-rank decomposition and activation sparsity.

Further, we examined the importance patterns of each input channel and singular value, as illustrated in Figure 3. These rank-aware sparsity patterns validate and underpin the core methodology of our framework, demonstrating its efficiency and effectiveness.

[Q2: Clarifying contribution] We respectfully disagree that our contribution is incremental. First, as discussed in Lines 51–64, one significant challenge in prior activation sparsity methods is the difficulty of predicting active channels. For instance, CATS utilizes the output of mlp.gate to identify active channels for sparsifying mlp.up and mlp.down, while GRIFFIN determines active channels based on output features from the pre-filling stage. In contrast, our method observes and exploits the inherent low-rank and sparse structures within the input features. This enables us to sparsify all linear layers without relying on prediction mechanisms, overcoming the limitations of prior methods. Specifically, CATS is restricted to sparsifying mlp.gate and mlp.down, and GRIFFIN applies sparsity only to MLP blocks. Our approach provides a broader and more effective solution to activation sparsity, making a substantial advancement over existing methods.

Additionally, the inherent low-rank and sparse structures within the input features contribute to significantly enhancing the achievable sparsity levels for non-ReLU-based LLMs. In comparison, methods like CATS and GRIFFIN are limited to achieving model-level sparsity ratios of only 22% to 33%, whereas our R-Sparse method achieves approximately 50% sparsity. Thus, we believe our R-Sparse makes a significant contribution to alleviate the main challenges of previous activation sparsity approaches, that is: (i) feasibility for non-ReLU based LLMs; (ii) difficulty in predicting active channels; and (iii) limited sparsity levels.

[Q3: Experiments with GELU activation] Great suggestion! We conducted additional experiments to evaluate our method on the Gemma-7B model. The results, presented in Table R1, show that our method significantly outperforms both CATS and GRIFFIN, achieving an average accuracy improvement of 21.76% and 4.62% at 40% and 50% sparsity, respectively.

Table R1 Comparison between R-Sparse and other baselines on common-sense reasoning tasks.

We sincerely thank Reviewer zFFb for the positive feedback and constructive suggestions. To address Reviewer zFFb’s concerns, we provide point-by-point responses below.

[Q1: Importance observation across different datasets and varying number of samples.]: Thanks for the question. We conducted additional experiments to assess the contributions of each input channel and singular value components across various datasets and different numbers of samples. The results are detailed in Section B.2 of the updated draft. Our findings consistently show that the primary contributions of importance are concentrated in the lower-right corner of the matrix. This supports the use of R-Sparse for accelerating inference without losing performance.

Details: To validate these observations across datasets, we examined diverse domains from the RedPajama dataset, including ArXiv, GitHub, StackExchange, and the original C4 dataset, ensuring a wide range of data diversity. Additionally, we evaluated varying sample sizes, ranging from 1 to 1024, and observed consistent patterns across all configurations.

[Q2: Explanation of varying importance observed across linear layers.]: Thanks for the suggestion. The varying importance mainly comes from the intrinsic low-rank properties of different linear layers. When the linear layer are more low-rank, the importance tend to centrarate only on the top-singular value components, results in a more sparse patterns in Figure 3 (i.e., the self_attn.k_proj) while for relatively higher rank, the importance patterns tend to distribute more uniformly across the vertical axis, (e.g., the mlp.up_proj). The layerwise low-rank properties aligns with previous investigations, such as Figure 4 in [1] and related with the learning dynamics of transformers(e.g., Section 4 in [2]).

[1] From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients.

[2] JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention.

[Q3: Relationship between sparsity ratio and final inference acceleration.]: Thanks for the question. Ideally, the acceleration equals to $\frac{1}{1-s}$ where s is the sparsity ratio. That is, 50% sparsity theoretically allows for a 2x speedup. However, in practice, due to the detailed memory access pattern, such as memory blocks not aligning perfectly with rows of the weight matrix, there is overhead associated with processing zero values. Additionally, activation sparsity doesn’t change the complexity of the scaled dot product in attention, as well as the KV cache overhead. Consequently, our implementation achieves a 1.4x speedup for 50% sparsity. For a more detailed analysis, we evaluate the latency of a single MLP block across different sparsity ratios. As shown in Table R1, the acceleration is gradually improved as the sparsity ratio increases. We evaluate on a single MLP block with input dimension of 4096 and hidden dimension of 11008, the latency is averaged by 80 runs after warm up running.

Table R1: Comparison of the latency across different sparsity ratios.

[Q4: Explanation of generation speed in Figure 6.]: Thanks for the question. For dense inference, the generation latency is primarily composed of three components: (i) the latency of prefilling, denoted as $t_{prefill}$, (ii) (decoding phase) the latency for computation and memory access of model parameters, $t_{weight}$, and (iii) (decoding phase) the latency for memory access of the KV cache, $t_{kv}(n_{prompt} + n_{generation})$ that linearly grows with the sequence length ($n_{prompt} + n_{generation}$). Therefore, the reported generation speed can be expressed as: $\frac{t_{prefill} + n_{generation}t_{weight}+n_{generation}t_{kv}(n_{prompt} + n_{generation})}{n_{generation}}$. In the early stages of decoding, the memory overhead of the KV cache is relatively small compared to that of the model parameters. As a result, the $\frac{t_{prefill}}{n_{generation}}$ term has significant impact on the end-to-end latency, but its effect diminishes as $n_{generation}$ increases (For generation length from 128 to 512), while for longer sequence length, the memory access cost of KV cache becomes more significant, as its size grows with the sequence length, leading to a reduction in the end-to-end generation speed.

We thank Reviewer T8xk for the valuable questions and suggestions. We elaborate more details of our method and provide point-wise responses in the following:

[Q1: Explanation of Table 1] Thanks for the question. R-Sparse 40% is not compared with CATS 22% and GRIFFIN 33%. Instead, we compare R-Sparse with other baselines under the same sparsity ratios (As discussed in Line 392), that is R-Sparse 40% v.s. CATS 40% and R-Sparse 50% v.s. GRIFFIN 50% where R-Sparse consistently outperforms both CATS and GRIFFIN. Additionally, we choose sparsity ratio of 22% for CATS and 33% for GRIFFIN for the reason that is the sparsity reported by the original publications where 50% mlp-level sparsity are applied and CATS only sparsifies mlp.gate and mlp.down while GRIFFIN sparsify the whole mlp modules. We’ve included this sparsity selection details in Line 370. And We’ve adjusted the lines in Table 1 in the updated draft to avoid misleading where R-Sparse 40% and CATS 40% are placed in adjacent lines as well as R-Sparse 50% and GRIFFIN 50%.

Additionally, we report the performance of various methods across different sparsity ratios in Figure 5, that provides a more thorough comparison and demonstrates our method consistently outperforms other baselines by a clear margin.

In certain cases, higher sparsity leads to improved performance. That’s because appropriate sparsity may help mitigate overfitting and enhance model generalization. This phenomenon has been demonstrated in lots of prior studies. Such as Figure 1 in SparseGPT[1], Figure 2 in Essential Sparsity[2] and Figure 4 in GRIFFIN[3].

[1] SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot.

[2] The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter.

[3] Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation.

[Q2: Sensitivity analysis of hyperparameters]: Thank you for pointing this out. We conducted a sensitivity analysis of the hyperparameters in Section 4.4 (A2 and A3), focusing specifically on the sparse-rank ratio $\rho$ for each linear layer. As discussed in Section 4.4 (A2), we compared R-Sparse with its sparse ($\rho=1$) and low-rank ($\rho=0$) counterparts, observing significant improvements, as presented in Table 3. In Section 4.4 (A3), we further compared the layer-wise $\rho$ values obtained through Algorithm 1 and the corresponding uniform $\rho$ assignment. As shown in Table 4, the adaptive $\rho$ consistently outperformed the uniform assignment across various sparsity ratios and tasks. These ablation studies provide a comprehensive evaluation of the effectiveness of R-Sparse.

[Q3: Comparison with Deja Vu]: Thanks for the question. Deja Vu requires training an additional predictor to identify active channels, whereas our comparison focuses exclusively on training-free approaches. To further address Reviewer T8xk’s concern, we conducted a comparison with Deja Vu using an oracle predictor—i.e., a predictor that perfectly identifies active channels with 100% accuracy. The results, presented in Table R2, demonstrate that our method outperforms Deja Vu (Oracle), achieving an average accuracy improvement of 0.51. Note that in realistic settings, the inaccuracy of Deja Vu’s predictor would further degrade its performance.

Table R2: Comparison of R-Sparse with Deja Vu (Oracle) under 50% sparsity on Llama-2-7B.

[Q4: Complexity analysis and running time comparison of R-Sparse]: Good suggestions. R-Sparse is a training-free approach, with its primary computational overhead stemming from the evolutionary search algorithm. The time cost of this search is linearly proportional to the population size ($P$), the total number of generations ($G$), and the number of samples used for perplexity evaluation ($N$), resulting in a complexity of $O(PGN)$. As discussed in Line 322, our experiments use a population size of 32, a generation count of 5, and 16 samples for perplexity evaluation. In a specific case, the search process requires approximately one hour on a single A6000 GPU for the Llama-2-7B model, which is negligible compared to the training-level computational cost.

[Q5: Writing improvement]: Good suggestions. The optimal sparse-rank ratio is defined as $\rho^* = \mathrm{argmin}{\rho} \mathcal{L}(f, \rho)$ where the loss function $\mathcal{L}$, represents the average perplexity computed over 16 randomly selected samples from the C4 training set, and $f$ denotes the original LLMs. To solve this optimization problem, we employ an evolutionary search algorithm, as outlined in Algorithm 1 of the PDF. In Algorithm 1, the variable $\alpha$ is a random variable used to control the crossover of individual solutions. It does not require optimization; instead, the sparse rank ratio $\rho$ will be optimized during each generation as shown in Line 10.

Thank you for the additional results. I have no further questions and will increase my rating accordingly.

We sincerely thank Reviewer LhWB for the detailed and insightful suggestions. To enhance the writing quality and address the concerns raised, we provide point-by-point responses below:

[Q1: Explanation about Figure 1.]: Thanks for the suggestion. We have revised the caption of Figure 1 in the updated draft to improve clarity and ensure it is easier to understand. The new caption is: “Contributions of each input channel and singular value components. The measurement metric is detailed in Section 3.3. Results are obtained from Llama-2-7B with 16 training samples from C4. Both the input channel and SVD components are sorted from small to large for better visualization.

[Q2: Explanation about the “Bias” term in Section 3.2.]: The bias term includes all the non-sparse components where $H_k < T_0$. Its definition depends on the value of $T_0$ rather than being strictly tied to $O$. If $T_0$ is greater than $0$, the non-sparse components include values that are both greater than and less than $O$.

We refer to this term as “bias” because the output $Y=HW_{down}^T$ can be interpreted as a weighted linear combination of the columns in $W_{down}^T$, where the coefficient of column $k$ equals $H_k$. For $H_k \geq T_0$, the calculation cannot be simplified, as each column is weighted by a different coefficient, and these are therefore defined as sparse components.

In contrast, for $H_k < T_0$, the columns satisfying $T_{i+1} \leq H_k < T_i$ share the same coefficient after the multi-phase ReLU activation function. These components can be viewed as a bias term, where the weighting is given by $\frac{T_j + T_{j+1}}{2}$.

[Q3: Appropriateness of paper title.]: Thanks for pointing this out. We’d like to clarify that the reason for using rank-aware activation sparsity is because for different linear layers, the activation sparsity is affected by its low-rank properties, thus we need to co-design the sparsity and low-rank decomposition to better preserve original functionality. We achieve this through an evolutionary search approach, which effectively balances these factors for optimal performance. We’re willing to adjust the title if Reviewer LhWB has further suggestions.

[Q4: Modification of Figure 3.]: Thanks for the careful reading. The modified Figure 3 is updated in the draft.

[Q5: More reference.]: Thanks for the suggestion. This work provides an interesting and solid hardware solution for sparsity-aware memory loading, that is orthogonal and provides a useful tool to further implement our methods on edge-devices. We’ve included the reference in the updated PDF.

[Q6: Definition of multi-phase ReLU.]: Great Catch! In our investigation, we set $T_{l-1}$ as the minimum value of input, to avoid scenarios when $x < T_{l-1}$. And the sparsity is defined as the ratios of $x < T_0$. In this way, for all $x < T_0$, the output will be the nearest discrete value of $\frac{T_i + T_{i+1}}{2}$, to better preserve the original functionality. We have clarified this explanation in the updated draft to avoid misleading.

Thank you for your kind response. There is one unclear part, and I would like to ask a question about it.

[Q2 and Q6]

• The authors said that "the sparsity is defined as the ratios of $x < T_{0}$", in Q6.
  • However, within this range, $x$ actually takes on the nearest discrete value rather than 0 in order to maintain the original functionality. Is it correct to define this as "sparsity"?
  • As requested in the original question, please provide an example to explain the situation. I understand the case where the minimum of $x$'s input is -2.
• Futhermore, the authors said that "all the non-sparse components where $H_{k} < T_{0}$", in Q2.
  • Is not this expression contradictory to the definition provided above?

Please clarify it. Thanks.

Thank you for the detailed feedback. The sparsity definition we use aligns with the spirit of previous network pruning works [1-3] where the sparse sub-network refers to the remaining sub-network, and the corresponding parameter values are unchanged after the pruning operation. The sparsity ratio is defined as the proportion of parameters that their values are changed after pruning (set to zero). Intuitively, the sparse components represent the critical parts of the network, while the non-sparse components correspond to the non-critical parts.

In our definition, the non-sparse components are those with $H_k < T_0$, as these typically have a lesser impact on the original functionality, making them the non-critical part. Conversely, the sparsity ratio measures the proportion of elements that are altered after applying the multi-phase ReLU (i.e., $x<T_0$). Thus, the two concepts are not contradictory.

Regarding the question, “x actually takes on the nearest discrete value rather than 0 in order to maintain the original functionality. Is it correct to define this as ‘sparsity’?” — Yes, this definition differs from the strict interpretation of sparsity, where non-critical values are explicitly set to zero. However, it aligns with a broader concept of sparsity. Here, sparsity refers to the effective reduction of non-critical elements, even if these values are adjusted to discrete values rather than being completely zeroed out, to preserve functionality.

For further clarification, we provide a conceptual example in Table R1. The table shows an input feature of size 10, with the values of each element reported before and after applying the multi-phase activation function.

Table R1, examples of values before and after multi-phase activation function. Where $T_0=0$, $T_1=-1$ and $T_2=-2$ and the minimum values of input features is -2. $s$ represents elements that belong to the sparse components, while (non-$s$) denotes the non-sparse components.

[1] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks.

[2] Rethinking the Value of Network Pruning.

[3] What is the State of Neural Network Pruning?